Researching existing approaches and join forces

[derhasi]

We should compare existing approaches and try to join forces on building "the drupal packagist", performant, complete, just awesome:

Existing projects and related issues

• Drupal Org issue
• https://github.com/reload/drupal-packages-generator
• Will Milton's packagist fork: https://github.com/winmillwill/packagist/tree/drupal
• https://github.com/winmillwill/drupal-parse-composer
• http://static.drupal-packagist.org/v0.2.0/packages.json

Sources for getting release information

• update.xml per project: http://updates.drupal.org/release-history/views/all
• All releases: https://drupal.org/files/releases.tsv (updated 4 times a day)
• https://www.drupal.org/api-d7/node.json?type=project_release
• Commit log: https://www.drupal.org/commitlog/feed (does not support ?page=...)
• Releases tagged 8.x: https://www.drupal.org/taxonomy/term/7234 (there's a feed as well)
• Releases tagged 7.x: https://www.drupal.org/taxonomy/term/103
• Module Index https://www.drupal.org/project/project_module/index

General ideas

• Parse commit log
• Listen to SAs to react as fast as possible!

[davidbarratt]

I was wondering if it would be possible to create a Drupal site that serves as a drupal packagist?

Composer gives a pretty good overview of what it's looking for:
https://getcomposer.org/doc/05-repositories.md

and Drupal.org has a REST API:
https://www.drupal.org/api

I think we could write a module to ingest the D.o. API, store what we need in custom entities, and then simply return the JSON that is required by Composer. (ok, it sounds simple when I write it out like that, haha)

[dawehner]

@davidbarratt
So you mean we could do exactly what @webflo did on http://drupal-packagist.webflo.io/

[davidbarratt]

yeah, except I think the site should be on Drupal rather than Symfony, but that might be pie in the sky hope. :)

[dawehner]

I would argue that there is no point at all ... drupal.org could host that instance or yeah reinvent the wheel.

[kasperg]

Nice job @webflo!

I think a fork of Packagist is great at this early point. On the other hand Packagist contains a lot of duplicated functionality - search for packages, showing packages information, user registration, publishing and maintaining packages etc. At best this is unneeded - at worst confusing.

As I see it the core of what we need is Composer repository - not a complete Packagist site.

Before this makes it into the drupal.org infrastructure I think we should consider our options. One approach to be to work with the Composer community to separate Packagist frontend from backend.

[webflo]

Thx, but the @winmillwill deserves the credit. I took his code, studied it and deployed it :)

I think Packagist is a good starting point. It´s very straight forward and does a good job to manage all package information and keep them up to date. The generated json files are static, there is no need to expose the web frontend if we move this into the d.o infrastructure.

Currently its just very convenient to view the actual package information and dependencies between modules.

[winmillwill]

The other thing to consider is that the composer user agent can (and does) use the search end point. Granted, this does not happen in the critical install and update use cases.

Still, the easiest thing is to just configure the http server to keep anyone from accessing confusing paths.

[derhasi]

Building a drupal packagist from scratch with Drupal would be a lot of work. I think using packagist as base is great, as it most of the functionality we need is already in there. The original project is the way composer is used to work with. With the web interface we've got some good starting point for "proofreading" and can double-check our composer.json's information.

@winmillwill are you hosting http://drupal-packagist.org/? If so, what code is it running on?

[winmillwill]

static.drupal-packagist.org is just an S3 bucket to which I've uploaded
metadata generated with my packagist fork from periodically spinning up
several instances with the code and crawling all the Drupal.org repos that
have 7.x releases.

My last commits to that fork were for experimental automatic update
functionality, but the updates rss feed is all downcase, but the repo URLs
are case sensitive, so there will need to be an additional provision for
keeping releases.tsv fresh, or just not supporting projects with stupid
names.
On 24 Dec 2014 09:57, "Johannes Haseitl" notifications@github.com wrote:

Building a drupal packagist from scratch with Drupal would be a lot of
work. I think using packagist as base is great, as it most of the
functionality we need is already in there. The original project is the way
composer is used to work with. With the web interface we've got some good
starting point for "proofreading" and can double-check our composer.json's
information.

@winmillwill https://github.com/winmillwill are you hosting
http://drupal-packagist.org/? If so, what code is it running on?

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

[derhasi]

Would you be fine with webflo's fork running on your drupal-packagist.org? I could see to move his code here for "community development". We should not rush on that, but for getting some drive in people actually using composer for Drupal projects, this could be an important step.

[winmillwill]

I am happy with the changes that @webflo has made to the packagist fork, and I'm happy to be available to help and glad for the interest. There is ongoing discussion about the details of representing package dependencies in drupal/parse-composer#.

The thing is that the drupal-packagist.org domain is just an AWS Route 53 DNS entry that doesn't point anywhere. Only static.drupal-packagist.org actually points at a service, and that's just an S3 bucket. If someone wants to pay for a server to run the packagist fork in this organization, I'm happy to hand over the domain.

I think we should just have a few http server rewrites for the front page and other potentially confusing paths so that it's clear that we don't support the normal packagist.org workflow (register, login, add a repo). This way we can leave it on and use the experimental auto update functionality.

Longer term, I think we can do as @kasperg suggests and write some pull-requests for upstream packagist so that our code can be a separate repo instead of a fork, and like @dawehner said, I think we should see about getting someone from drupal.org and/or the Drupal Association interested so that this can become the official way to manage drupal packages.

In any case, I think this issue is resolved unless there are more projects/approaches that we need to consider?

[derhasi]

@webflo just pushed that code to this project. So we have no github fork anymore and can use the repo search :) We hope to get some more visibility from that too.

@winmillwill, If you want we could move drupal-parse-composer too to the org, as it is a crucial part for the packagist. I'd add you to the organisation as well, if you are interested in that too :)
I agree with you on the other points and therefore should create separate issues for those.

[derhasi]

@winmillwill ++ :)