Support any composer-type repository

[Elendev]

I tried to use this plugin to connect to Packagist (ok) and to Drupal 8 composer repository (ko).

After a small investigation, the plugin tries to download the file packages/list.json and this file seems to be a specificity of packagist.org.

The packages.json file contains the links to the provider files. The provider files contains the list of the packages and their signatures.
The provider files are the ones used by composer.

The packages/list.json has the list of the available packages of packagist.org but it looks like it's not used at all by composer, nor it's provided by Drupal 8's repository.

To solve this issue, we should :

• update the method generatePackagesJson in the class ComposerProxyFacetImpl to only use the providers
• don't use the constant PROVIDER_JSON in the ComposerProxyFacetImpl class, instead use the value provided by the packages.json file
• handle the case when a provider is not available anymore (it seems to be the case for old versions of providers)

Some technical details

The entry point is the packages.json file.
Examples :

• for Packagist : https://packagist.org/packages.json
• for Drupal : https://packages.drupal.org/8/packages.json

The packages.json file contains a providers-url and a provider-includes entries.

The providers-includes entry list all the files containing the list of repositories.
Examples :

• for Packagist : p/provider-2018-01$%hash%.json with hash 9e9bb7cf12c2f46329fde5008d3ce061405ff7d1e88150e64e98fc4df8b46b7b that will be transformed to https://packagist.org/p/provider-2018-01$9e9bb7cf12c2f46329fde5008d3ce061405ff7d1e88150e64e98fc4df8b46b7b.json
• for Drupal : drupal/provider-2018-1$%hash%.json with hash f0bc8344a8c5af51478a86cc24911a2e36441b92a88085f534445b2531aabb7b that will be transformed to https://packages.drupal.org/8/drupal/provider-2018-1$4d04d5677c5045740aa28bd346e9889c2bd0d5ea23a2c71ef132585be9952777.json

The providers-url contain the format to use to download the description of a specific package.
The required informations (%package% and %hash%) are available in the provider file described above.
Examples :

• for Packagist : /p/%package%$%hash%.json, with the package faustbrian/ark-symfony and its hash 52b4cbe41a74b0c787f853f9c5522caa764c4e20fe660ade907af46d35f5c0f2 : https://packagist.org/p/faustbrian/ark-symfony$52b4cbe41a74b0c787f853f9c5522caa764c4e20fe660ade907af46d35f5c0f2.json
• for Drupal : /8/%package%$%hash%.json, with the package drupal/token and its hash 0e905c202a67a1b2a08bbd6da858b8af93c01ec6d06dd8f3488b318acfe5680c : https://packages.drupal.org/8/drupal/token$0e905c202a67a1b2a08bbd6da858b8af93c01ec6d06dd8f3488b318acfe5680c.json

[fjmilens3]

Thanks for writing this up. This is actually a known issue with the implementation as we're essentially trying to get a working MVP out there for the community to build on rather than trying to make it perfect. What you describe very much parallels many of our early learnings in reverse-engineering/studying this format; there are a couple of primary reasons we didn't use the approach you described as the MVP, so here's a bit of background:

One reason is that there's some significant overhead in trying to process and rewrite those nested layers of provider files, and it's not something that our current formats infrastructure is necessarily the best match for. For example, Nexus has to rewrite download links in order to redirect the client through the repository manager, but rewriting one of the JSON files changes its hash, which means that a change at any "leaf" JSON file will propagate up the entire chain. Also consider that we don't know what files will be requested, so we cannot do the rewrite in advance, thus necessitating some sort of mirror or sync mechanism to keep things matching the upstream repository. This is not unheard of, but it's a poor fit for the normal mechanisms of Nexus proxy repositories. In many respects, the sort of design that went into making Composer a more challenging target for a man-in-the-middle attack at the index-file/registry level also makes it more challenging for us to effectively go in and change things.

The other reason is that flattening the list of providers also helps us in other ways, such as with implementing the group merging feature of the Nexus repo. Group repositories have to merge the contents of all member repositories, and in different repositories with different provider/nesting structures and filenames, the group merge could quickly get a lot more expensive and certainly won't be something that we could do on a lazy basis. We have other formats that have to do this kind of more complicated metadata merging across different repositories and while we could figure it out, it would have taken a lot longer than we had available to devote to what's essentially a spare-time project done under the Sonatype Community aegis.

That said, this is something that's been on my radar that I'm hoping to get around to once we have a minimal implementation of proxy, hosted, and group working for Composer. Until then, I'd say that our Composer proxy also requires the additional API method provided by Packagist at this time, though we're always happy if someone wants to start picking up one of these issues and coming up with possible paths forward.

(One suggestion I might have would be to see how difficult in terms of implementation and how expensive in terms of performance it might be to synthesize the equivalent of a list.json file by traversing the providers as you describe, using that as the reference to build the flattened JSON file that proxy is currently serving up as a substitute. That said, that's not a complete solution because you will need the sha values in order to access the packages hosted by Drupal even though we don't need those when accessing the same files on Packagist, but it might be a place to start an investigation.)

[fjmilens3]

I've given this a bit more thought and another option might be to rely on the same "hack" that we currently use when building the provider file, namely that if the sha is blank (but the key is still present) the composer client effectively ignores the sha and fetches the file without an associated hash in the filename.

So long as this holds we might be able to minimize the rewrites we have to coming up from the leaf nodes, but we'd have to have a way to back-associate the sha values in order to proxy the targeted file. I think we'd also have to still rewrite the provider-includes section, but that's not that much worse than what we currently have to do for other formats.

Note that I have not come up with a POC for this and likely won't for some time. Also note that this solution doesn't take into account the group merge behavior, but perhaps we could combine the approach with a "flattening" sort of transformation like we're currently doing. Once I get done with hosted (and perhaps group) I may investigate this further if nobody else has grabbed it by then.

[Elendev]

One suggestion I might have would be to see how difficult in terms of implementation and how expensive in terms of performance it might be to synthesize the equivalent of a list.json file by traversing the providers as you describe, using that as the reference to build the flattened JSON file that proxy is currently serving up as a substitute

That's exactly what I was thinking in the first place. I don't think we need at first a "copy" of the way composer is working, but rather a way for this plugin to use other composer repositories (particularly the Drupal one)

Note that I have not come up with a POC for this and likely won't for some time

No problem, we've come with a temporary solution with this issue. I've forked the repo and wanted to fix this issue, but I've discovered that I don't have enough knowledge with Nexus nor with Java, neither do I have enough time to learn everything to fix this :-/

[DarthHater]

@Elendev if you got it working, post up a PR. We can always work together to try and get it done together :)

[Elendev]

@DarthHater unfortunately I just took a look at the code but I wasn't able to find time to start a PR about this, so feel free to start working on this issue :-)

[fjmilens3]

Some improvements will be made as part of #22 but will not solve the matter for all repositories. In particular, I think we have to make some other significant changes (supporting sources) to be able to get it working with the Drupal repositories, which we'll have to leave for a later date. Refer to the conversation on that PR for details.

[CelesteGH]

Still no news ?

[ludwis]

Hi, will this issue get some traction any time soon?