Enhance support for other Composer repositories #22
Conversation
|
Wanted to chime in that this branch at least initially seems to have made the Wordpress repo at https://wpackagist.org work. Thank you @fjmilens3 for your work on this! 🙏 |
|
First thank you @fjmilens3 for your work ! I've tried this PR and I have encountered some issues. For my tests I use a local nexus with the last version of Drupal (8.5.5), I've downloaded the ZIP file available on the website. "repositories": [
{
"packagist.org": false
},
{
"type": "composer",
"url": "http://localhost:8081/repository/Packagist/"
},
{
"type": "composer",
"url": "http://localhost:8081/repository/Drupal8/"
}
],When I try to run You are running Composer with SSL/TLS protection disabled.
Loading composer repositories with package information
Warning: Accessing localhost over http which is an insecure protocol.
Updating dependencies (including require-dev)
Package operations: 46 installs, 24 updates, 0 removals
- Updating symfony/class-loader (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/debug (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/console (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/dependency-injection (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating paragonie/random_compat (v2.0.11 => v2.0.17): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/http-foundation (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/event-dispatcher (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/http-kernel (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/routing (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/serializer (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/translation (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/validator (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/process (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Installing doctrine/reflection (v1.0.0): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Installing doctrine/event-manager (v1.0.0): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating doctrine/collections (v1.3.0 => v1.5.0): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating doctrine/cache (v1.6.1 => v1.7.1): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Installing doctrine/persistence (v1.0.0): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating doctrine/inflector (v1.1.0 => v1.3.0): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating doctrine/common (v2.6.2 => v2.9.0): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating zendframework/zend-stdlib (3.0.1 => 3.2.0): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating zendframework/zend-escaper (2.5.2 => 2.6.0): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating zendframework/zend-feed (2.7.0 => 2.10.2): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating zendframework/zend-diactoros (1.4.1 => 1.8.1): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Installing symfony/css-selector (v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Installing behat/mink (dev-master): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Installing instaclick/php-webdriver (1.4.5): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Installing behat/mink-selenium2-driver (dev-master): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Updating symfony/yaml (v3.4.11 => v3.4.12): Downloading (100%)
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
- Installing squizlabs/php_codesniffer (2.9.1): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
[InvalidArgumentException]
Package drupal/coder-8.2.12.0 must have a source or dist specified
update [--prefer-source] [--prefer-dist] [--dry-run] [--dev] [--no-dev] [--lock] [--no-custom-installers] [--no-autoloader] [--no-scripts] [--no-progress] [--no-suggest] [--with-dependencies] [--with-all-dependencies] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [-i|--interactive] [--root-reqs] [--] [<packages>]...The generated package file (http://localhost:8081/repository/Drupal8/p/drupal/coder.json) is available in this gist: https://gist.github.com/Elendev/1aed2fd0a6d114cfd95608466e5c3739 It's interesting to note that the dependency in the |
|
@Elendev, as mentioned on my other reply, I'm not sure when I'm going to have more time to really look at this. I'm hoping I will sometime in the next couple of weeks but I can't make any guarantees. One thought is to perhaps find the upstream provider file, for example https://packages.drupal.org/8/drupal/coder$918ecf2148ea212e14f19fab124f2c8ca6ef3f19c79ddbda429e1ec282be7748.json (where it is right now, but that can always change) and do a semantic diff on the JSON relative to what Nexus generates. That might tell us if in fact we are doing something in the JSON that's messing up Composer or if there's something else in play. Additionally, note that we don't support sources, only dists, in the plugin's current implementation, so if something doesn't have a Given @dfwarden's comment that this seems to at least be working with wpackagist.org, I'm also interested in characterizing the scope of the problem. Depending on the magnitude of the additional work it may be something I end up breaking out into its own issue to resolve independently. (That said, I would really like to get this to work for Drupal at the moment.) |
|
I just Nexus for Java development (day job) and I love what I can do with it. I stumbled across this repo then this PR. I am a Drupal developer at night and have had issues with packages for Drupal plus other dependencies where I want to start having control over where I pull dependencies like I do with Java. This PR is great and maybe I can help get it to work with Drupal's package server. |
|
I compiled the plugin and installed it with the Karaf console on the latest Nexus (3.13) and it will not start the UI with the bundle running. I do not get a whole lot of useful messages in the Karaf console. I will have to see what logs nexus has I can tail, but it will be later as I have to switch back to working on some Drupal sites. |
@taz77, you actually need to be running against an older version of NXRM in order to use this plugin right now. :) @jlstephens89 has a PR up to update us to the latest NXRM release in #28 that should help with that matter at least. |
|
@taz77 @fjmilens3 #28 is now merged - feel free to grab a copy and give it a go |
|
@Elendev, I finally had some time during our Improvement Day (think "20% time" but once every two weeks instead) here and I'm able to reproduce what you're seeing. The exact steps for anyone following along:
I've gone ahead and diffed the JSON between each. So far, the major differences are that:
Other findings:
My suspicion is that for whatever reason it's using source distributions and not picking up any available dists, and since my NXRM implementation doesn't support source distributions, composer fails as it can't find a source, at https://github.com/composer/composer/blob/158e1c95da02cc0b932de74f9a09a1c7b6cf654f/src/Composer/Downloader/DownloadManager.php#L199. I haven't set up xdebug to watch what it's doing for version resolution yet to confirm this, however. I'm going to experiment a bit more with this today, as if it turns out we do need to support sources to get Drupal to work, I'm going to change this particular issue and PR to be more targeted toward repos like wpackagist so we can get this functionality baked in. I'd open another issue and PR for Drupal in particular, centering around getting sources working and cleaning up some of the renaming that goes on. Implementing that will be a rather major (and potentially somewhat destabilizing) undertaking and I don't want to conflate it with these changes if possible. |
fjmilens3
changed the title
|
Update: I'm starting a new job next week and likely won't be contributing much more to this particular project, though I may try to pop in from time to time. My intent is to get the improvements here merged in as my last bit of work on it, but there is still outstanding work to be done (both on this and on other issues). It sounds like this will at least help by getting it working for wpackagist and so on, but further work needs to be done to support the Drupal repositories and their source packages. |
|
Can i help here. I really would love to have this feature in this repository so i can use at my own. :) |
|
Thanks for the contribution! Before we can merge this, we need @fjmilens3 to sign the Sonatype Contributor License Agreement. |
|
We need to add the proxy " https://packages.drupal.org/8" in addition to the classic proxy "https://packagist.org/". We really would love to have this feature in this repository ! How can we validate this PR ? |
|
Thanks for the contribution! Unfortunately we can't verify if the committer(s), Frederick John Milens III fmilens@sonatype.com, signed the CLA because they have not associated their commits with their GitHub user. Please follow these instructions to associate your commits with your GitHub user. Then sign the Sonatype Contributor License Agreement and this Pull Request will be revalidated. |
|
I had to manually resolve some things that couldn't automatically merge but I was able to rebase this branch on to master and get a working 0.0.9-SNAPSHOT version of this plugin, in case other folks are following this PR and are wondering if it can work with newer-ish (3.27.0-03) versions of Nexus. |
|
thank you , we use the code to proxy https://mirrors.huaweicloud.com/repository/php/ work well.the version is tag 0.0.8, nexus version is 3.25.1-04, this is a useful feature ,merge to the master will be a good idea. |
|
@liaoguode can you provide the jar file of the plugin? |
I'm sorry，the jar is the product include private information.I only use the pr code to the version.You should notify use the |
|
Hi. We were actively looking for the functionality this plugin provides, but were very sad to see the great work done here have been stuck for more that 2 years and not merged back to We were wondering if there are any ways we could contribute this new branch back to this repository for revision, acceptance and merge. How can we do that? Thanks. |
Hi Did you manage to make it work with Drupal or an other repo ? Thanks |
|
We're also affected by this and this PR seems to be getting quite some attention. Is there any way to drive this forward? Are you fine with somebody taking over @fjmilens3? Sorry for doing it again, but maybe somebody from the community can help out too? @acockrell @bradbeck @csgillespie @DarthHater @davidkarlsen @mpiggott @whyjustin |
|
I'm not sure about the state of this PR and sorry for tagging you directly, but since there was recent activity I just could not resist :) Is this also maybe something you might want to look into @bhamail? |
|
@herndlm No worries with tagging me directly. Unfortunately, I can't say I have cycles (nor expertise) to pick up this PR. I will post a comment on an earlier request to contribute an updated version - which might be the best approach at this point. |
@flagsibh I'm just seeing your post about this. Sorry for the slow reply. Given the original PR has suffered some bit-rot, it might be best for you to create a new PR. I would suggest you first merge the latest from master. I'm no expert on Nexus formats, nor Composer, but I'm happy to review and merge a PR if the community can do some validation/testing that the PR works "good enough". |
|
As of today (2022-08-04) https://wpackagist.org does not work. When requesting Nexus: OSS 3.41.0-01 |
Adds support for other repositories, not relying on the conventions of the main packagist repo or the availability of the custom
list.jsononly available from that repo.Under this approach, instead of going remote on ourselves to fetch the
list.json, we fetch thepackages.jsonand fetch the contents of all referenced provider-includes. We generate apackages.json-like file with the original providers url and the associated hashes for the upstream provider files, which are then accessed and referenced when determining the url for the provider file for a particular vendor/project.For now I've limited my initial testing to the Drupal repo at https://packages.drupal.org/8.
There are likely some serious scalability issues introduced as part of this PR that I want to come back and resolve, either in this PR or in a subsequent PR. I'm less concerned about the overhead of the initial build/merge and far more concerned about reading the rather large resulting JSON into memory for each provider.json request. We might be able to reduce the memory overhead significantly by parsing the content using the token-based parser and only grabbing the portions we need, though that will require additional work (either here or in an additional PR).
Update: Additional changes are required for full support of the Drupal repositories (see conversations below). However, since I'm leaving Sonatype for my new job at the end of next week, I'd like to get these particular improvements merged into master.