Add a option to install and update command to controll whether loads provider listings

[xjchengo]

Most of time, the provider listings loading process needs a lot of time under my network.

[4.3MB/0.02s] Reading ./composer.json
[4.4MB/0.02s] Loading config file /home/vagrant/.composer/config.json
[4.5MB/0.02s] Loading config file /home/vagrant/.composer/auth.json
[4.5MB/0.02s] Loading config file ./composer.json
[5.0MB/0.03s] Executing command (CWD): git describe --exact-match --tags
[5.3MB/0.04s] Executing command (CWD): git branch --no-color --no-abbrev -v
[5.3MB/0.04s] Executing command (CWD): hg branch
[5.3MB/0.05s] Executing command (CWD): svn info --xml
[6.8MB/0.07s] Reading /home/vagrant/.composer/composer.json
[6.8MB/0.08s] Loading config file /home/vagrant/.composer/config.json
[6.8MB/0.08s] Loading config file /home/vagrant/.composer/auth.json
[6.8MB/0.08s] Loading config file /home/vagrant/.composer/composer.json
[6.8MB/0.08s] Loading config file /home/vagrant/.composer/auth.json
[6.8MB/0.08s] Executing command (CWD): git describe --exact-match --tags
[6.8MB/0.09s] Executing command (CWD): git branch --no-color --no-abbrev -v
[6.8MB/0.09s] Executing command (CWD): hg branch
[6.9MB/0.10s] Executing command (CWD): svn info --xml
[7.6MB/0.13s] Loading composer repositories with package information
[7.9MB/0.16s] Downloading https://packagist.org/packages.json
[8.0MB/23.93s] Downloading https://packagist.org/packages.json
[8.0MB/25.05s] Writing /home/vagrant/.composer/cache/repo/https---packagist.org/packages.json into cache
[8.0MB/25.07s] Installing dependencies (including require-dev)
[8.4MB/25.08s] Reading /home/vagrant/.composer/cache/repo/https---packagist.org/p-provider-2013.json from cache
[8.4MB/25.08s] Reading /home/vagrant/.composer/cache/repo/https---packagist.org/p-provider-2014.json from cache
[8.4MB/25.08s] Reading /home/vagrant/.composer/cache/repo/https---packagist.org/p-provider-2014-10.json from cache
[8.4MB/25.08s] Reading /home/vagrant/.composer/cache/repo/https---packagist.org/p-provider-2015-01.json from cache
[8.4MB/25.12s] Downloading https://packagist.org/p/provider-2015-04$e116192acc59f914d7df3be6e373553528705eea2f18244f6309af5121b43aa0.json
[10.2MB/157.81s] Downloading https://packagist.org/p/provider-2015-04$e116192acc59f914d7df3be6e373553528705eea2f18244f6309af5121b43aa0.json
[48.9MB/180.26s] Writing /home/vagrant/.composer/cache/repo/https---packagist.org/p-provider-2015-04.json into cache
[48.2MB/180.31s] Reading /home/vagrant/.composer/cache/repo/https---packagist.org/p-provider-2015-07.json from cache
[42.2MB/180.34s] Reading /home/vagrant/.composer/cache/repo/https---packagist.org/p-provider-archived.json from cache
[42.0MB/180.37s] Downloading https://packagist.org/p/provider-latest$906f94c57e4950afdcb384142391fadcd3fe7569f148f788b70b9f10ee0077c2.json
[47.1MB/182.60s] Writing /home/vagrant/.composer/cache/repo/https---packagist.org/p-provider-latest.json into cache
[47.1MB/182.67s] Downloading http://packagist.org/p/vlucas/phpdotenv$919fe0a419aa507aa84bfc58a49a02ac1366a1a6c9fd5fb3ba990bf12b795aa1.json
[47.3MB/183.35s] Writing /home/vagrant/.composer/cache/repo/https---packagist.org/provider-vlucas$phpdotenv.json into cache
[47.7MB/183.48s]   - Installing vlucas/phpdotenv (v1.1.0)
[48.2MB/183.54s] Reading /home/vagrant/.composer/cache/files/vlucas/phpdotenv/732d2adb7d916c9593b9d58c3b0d9ebefead07aa.zip from cache
[48.2MB/183.55s]     Loading from cache
[48.2MB/183.55s]     Extracting archive
[48.2MB/183.56s] Executing command (CWD): unzip '/home/vagrant/Projects/composer-test/vendor/vlucas/phpdotenv/de2d6271ee426d5d56ae484f2bbbfe0b' -d '/home/vagrant/Projects/composer-test/vendor/composer/feb0d38c' && chmod -R u+w '/home/vagrant/Projects/composer-test/vendor/composer/feb0d38c'
[48.3MB/183.91s] 
[48.3MB/183.91s]     REASON: Required by root: Install command rule (install vlucas/phpdotenv 1.0.0|install vlucas/phpdotenv v1.1.0)
[48.3MB/183.91s] 
[47.8MB/185.97s] Writing lock file
[47.8MB/185.97s] Generating autoload files
[47.8MB/186.03s] Memory usage: 47.84MB (peak: 55.11MB), time: 186.03s

Can we add a option to install and update command to indicate whether we need loads provider listings? In order to make this work, packagist needs to add providers-lazy-url like toran proxy does.

[Seldaek]

In packagist's case, the provider listings help cache things because then composer knows if the hash of a file didn't change it doesn't have to redownload it, if we didn't have the listings it would mean every package you require has to be downloaded (indeed like in toran proxy). That said with the growing amount of packages maybe we need to revisit this and use a process like toran.. It needs to be investigated whether it'd be faster or slower on average. It's hard to know because with the current packagist model, if you run update twice in a row, the second one will most likely not download much since it has everything in cache.

If we had parallel (like with the react PR or similar) downloads then it might be better to always hit packagist for every package and rely on e-tags or something to save bandwidth. Right now though that is not possible.

[xjchengo]

From my view of source, this mechanism doesn't affect the cache of dist files. It affects the cache of providers data like provider-composer$composer.json. Using providers-lazy-url makes it cannot use providers data cache but can use dist files cache too. But data like provider-composer$composer.json changes quickly. Every time I push a commit, it will change because the reference of dev-branch changes. If we remove reference of dev-branch from provider-vendor$package.json, the providers data cache will be more useful. This sacrifices the time of people using dev-branch to make normal users more quick. Or can we seperate provider-vendor$package.json into two files, one is stable and another is dev ?

[Seldaek]

@xjchengo I hope you can try this, I added an experimental flag now so you can enable lazy mode for packagist, I would love to get feedback from you on how well it works. You will need to run composer self-update to get the latest version.

First it would be great if you could run: composer update --dry-run --profile twice in a row and paste the last line of both runs here (this is our control run, plus control with primed cache).

To enable it, run composer config --global -e then make sure you have packagist repo set up like this:

{
    "repositories": {
        "packagist": {
            "type": "composer",
            "url": "https://packagist.org",
            "force-lazy-providers": true
        }
    }
}

Then you can again run composer update --dry-run --profile twice and paste the last line for both runs (first one will redownload all providers, second run should only send If-Modified-Since requests to packagist but use content from cache).

For me the 4 runs look like this:

current, no cache, [147.7MB/35.90s] Memory usage: 147.72MB (peak: 496.74MB), time: 35.9s
current, primed cache, [147.7MB/12.76s] Memory usage: 147.72MB (peak: 496.74MB), time: 12.76s

lazy, no cache, [96.0MB/33.98s] Memory usage: 96.01MB (peak: 445.03MB), time: 33.98s
lazy, primed cache, [96.0MB/30.28s] Memory usage: 96.01MB (peak: 445.03MB), time: 30.28s

So it doesn't seem like a big win for me to enable this, but I do have a good connection to packagist so I am curious to see how it goes for people with other connection quality.

[xjchengo]

@Seldaek
My results:

normal, no cache, [106.8MB/197.90s] Memory usage: 106.82MB (peak: 167.13MB), time: 197.9s
normal, primed cache, [107.1MB/10.01s] Memory usage: 107.11MB (peak: 167.42MB), time: 10.01s
lazy, no cache, [58.9MB/134.75s] Memory usage: 58.88MB (peak: 119.28MB), time: 134.75s
lazy, primed cache, [58.6MB/123.83s] Memory usage: 58.63MB (peak: 118.91MB), time: 123.83s

These results were gotten when I connected to a vpn with best overseas communication link quality from my country.
If I use composer directly without vpn, I will be sticked and most of time connot succeed to complete composer command. Because of slowness and error prone of composer install & update, installing or updating packages managing vendor with composer is nightmare to many PHP programmer around me.
What can I do to improve this situation?

[hirak]

I tried in Japan.

repository = laravel/laravel

normal, no cache, [147.3MB/150.86s] Memory usage: 147.32MB (peak: 176.51MB), time: 150.86s
normal, primed cache, [147.3MB/10.58s] Memory usage: 147.32MB (peak: 176.51MB), time: 10.58s
lazy, no cache, [96.5MB/198.46s] Memory usage: 96.48MB (peak: 125.67MB), time: 198.46s
lazy, primed cache, [96.5MB/153.98s] Memory usage: 96.48MB (peak: 125.67MB), time: 153.98s

[stof]

looks like the lazy way does not improve things but actually makes them slower

[Seldaek]

@xjchengo @hirak can you two please try it again but this time the lazy mode use (note .com and not .org):

{
    "repositories": {
        "packagist": {
            "type": "composer",
            "url": "https://packagist.com",
            "force-lazy-providers": true
        }
    }
}

You need the latest composer phar again so self-update. The .com uses cloudflare and has long term edge caching enabled which for now will cause delays when packages get updated, but if it seems to help I will migrate .org there too and add API calls to do cache busting at the cloudflare level when we update package metadata.

This should make the if-modified-since requests a lot faster but there is still the overhead of creating an ssl connection every time so I'm not expecting extremely low numbers. Also curious to see what numbers you get on the regular non-lazy repo using the .com URL (just remove force-lazy-providers to try it).

[hirak]

@Seldaek 👍 In my environment, packagist.com is 2x faster than packagist.org. Excellent!!

I think If-Modified-Since & 304 Not Modified is not the best way. IMS must request to packagist.org.

Fast:
client(Japan) <-> cloudflare (normal request)

Slow:
client(Japan) <-> packagist.org (normal request)
client(Japan) <-> cloudflare <-> packagist.org (with If-Modified-Since)

settings	result
https://packagist.com, lazy: false	[208.3MB/35.49s] Memory usage: 208.33MB (peak: 249.66MB), time: 35.49s
no-lazy, 2nd challenge	[208.3MB/17.00s] Memory usage: 208.31MB (peak: 249.61MB), time: 17s
https://packagist.com, lazy: true	[158.1MB/32.47s] Memory usage: 158.09MB (peak: 199.42MB), time: 32.47s
lazy, 2nd challenge	[158.1MB/30.90s] Memory usage: 158.09MB (peak: 199.38MB), time: 30.9s

$ curl --head --location https://packagist.org/packages.json --write-out " dns: %{time_namelookup}\n tcp: %{time_connect}\n tls: %{time_appconnect}\n redirect: %{time_redirect}\n total: %{time_total}"
HTTP/1.1 200 OK
Server: nginx
...
 dns: 0.005
 tcp: 0.358
 tls: 1.382
 redirect: 0.000
 total: 1.689

$ curl --head --location https://packagist.com/packages.json --write-out " dns: %{time_namelookup}\n tcp: %{time_connect}\n tls: %{time_appconnect}\n redirect: %{time_redirect}\n total: %{time_total}"
HTTP/1.1 200 OK
Server: cloudflare-nginx
...
 dns: 0.005
 tcp: 0.016
 tls: 0.285
 redirect: 0.000
 total: 0.851

[Seldaek]

Regarding if modified since, this is handled purely by cloudlfare on /p/package/name.json urls, as I enabled long edge cache on those. The initial /packages.json is always revalidated though so it's not the best file to test with. Anyway glad to see cloudlfare helps already. I'll try to move forward with that then I think, for the if modified since approach we will need more changes to make it viable but that's for later then. Thanks for testing.

[hirak]

@Seldaek If-Non-Match and weak Etag are supported by cloudflare. It's worth a try.

• https://en.wikipedia.org/wiki/HTTP_ETag
• https://support.cloudflare.com/hc/en-us/articles/218505467-Does-CloudFlare-support-ETag-headers-

[Seldaek]

The problem is it seems it doesn't help enough as is.. Using e-tags or last modified date is the same, it's about the amount of requests and the time they take to get bootstrapped. I think once we have persistent curl connections then we can switch to the lazy mode and do If-Modified-Since, but this will of course have to be checked.

[hirak]

@Seldaek Sorry, you are right. Did you change any settings for packagist.com?
I tried measuring now, it was crazy fast. "force-lazy-providers" + CDN is effective.

$ cat ~/.composer/config.json
{
    "config": {
    },
    "repositories": {
        "packagist": {
            "type": "composer",
            "url": "https://packagist.com",
            "force-lazy-providers": true
        }
    }
}
$ cd path/to/laravel/laravel
$ composer update --dry-run --profile
...
[89.1MB/10.61s] Memory usage: 89.13MB (peak: 153.22MB), time: 10.61s

[Seldaek]

I didn't touch this in a while, but to be honest the CloudFlare approach isn't really feasible I think as their terms of services indicate it should only be used for web pages and they can shut us down without notice. I can't risk having our DNS provider shutting us down like that :/ I looked at other CDN providers though and there might be viable options but I check this further after summer holidays I think. Maybe even after introducing the curl-based parallel requests, as that would benefit already a lot and reduce the issue of latency.

[Seldaek]

Fixed by composer 2