Packagist.org does have mirror sites?

[netorica]

I live in a country with terrible internet connection and I don't know if this is implemented already on the packagist.org If not then here's my question

does packgist.org has mirror sites so composer can download updates closer to my location? thanks :)

[alcohol]

No mirror sites at the moment unfortunately.

[netorica]

does anyone have plans funding for mirror sites for packagist? almost all PHP framework has its own web server and uses packagist to store their framework dependencies

[alcohol]

Unlikely. There is currently also no mechanic to properly sync multiple packagist mirrors. It would require some additionality functionality to push changes from the main packagist repo to all mirrors. Alternatively could build a pull sync mechanism into the mirrors, but then you'd risk it being contiously slightly out of sync. The frequency of updates processed by packagist is relatively high, which makes a mirror setup fragile at best.

[stof]

thus, this would only be about download package metadata, because unlike the npm registry, Packagist does not store the code for packages. Code is downloaded from github/bitbucket/...

[mousavian]

I have set up some mirrors listed here in Packagist Mirror List (currently in Singapore & Frankfurt). I'll appreciate if anyone can test and give me some feedback.

--
PS: shut down already.

[hirak]

Japan mirror is here.
https://packagist.jp

$ composer config -g repositories.packagist composer https://packagist.jp

[stof]

@Seldaek a solution for this might be to use Fasly, Cloudflare or an equivalent solution in front of Packagist. What do you think about this ?
Regarding costs, Fastly provides their service for free for many open-source projects (see https://www.fastly.com/open-source), so it might be worth contacting them

[ryanaslett]

We use fastly in front of packages.drupal.org and its really fantastic. Fastly is a great company to work with.

[hanoii]

I think this has become a must right now, and wonder what's holding it. Sometimes performance of composer update --with-dependencies is excruciatingly slow. For an active development project where you sometimes needs to update composer packages mor or less often, it does take a hit the downloads are so slow. Fastly should be an awesome improvement and probably not to hard to setup.

[stof]

@hanoii there is a north-american mirror now. and this is precisely what's causing issues currently (see composer/composer#6342)

[hanoii]

@stof thanks.. I am not in NA though. I am in SA... although I might probably be hitting that mirror anyway.. download speed is really bad though and I normally have good DW speed from the US.

[stof]

@hanoii but maybe your computer connects to the north-american server rather than the european one

[hanoii]

@stof very likely. Just wondering, not because of this issue of just now, but the download speed it is normally very slow.

[awesee]

In China, we use https://packagist.phpcomposer.com

$ composer config -g repo.packagist composer https://packagist.phpcomposer.com

[webysther]

Hey @hirak! Mirror in Brazil too https://packagist.com.br/.

[webysther]

@stof Packagist.org maybe create a page to point to alternative mirrors and how create using the crawler of @hirak, this solution is very simple and help people of diverse locations of world.

Another solution is make composer swicth between regions, a JSON with regions and a option to point to some region:

$ composer config -g repositories.mirrors.priority br us jp cn

[webysther]

Hey guys, i made some advances about mirror:

https://github.com/Webysther/packagist-mirror
https://github.com/Webysther/packagist-mirror-docker

Is very early stage but support concurrently, use really low resources, support fallback to main mirror and use more than one mirror. Is used on https://packagist.com.br

Suggestions, tips and help will be welcome.

[ryanaslett]

What if we tried to use git as our mirroring and delta distribution mechanism?

Every time packagist runs the symlinkdumper and generates a new packages.json and all of the provider files on the filesystem, it could then commit those files to a packages metadata git repo.

Packagist.org could run it's own git daemon, or push the commits to github, and if somebody wanted to setup a mirror, they would just need a clone of that repo, and periodically do git pulls from the metadata repo.

This is similar to what cocopods does (https://github.com/CocoaPods/Specs), however we dont really need every end user to have a copy of this repo, just folks who would like to set up a mirror. (because of things like this: CocoaPods/CocoaPods#4989 (comment))

Even if there were 10 mirrors polling every minute, Thats only 100,000 pulls a week, and all of those would be small deltas. If mirrors were willing to be 5 or 10 minutes behind, that could be significantly fewer.

Anyhow, just a thought.

[webysther]

@ryanaslett As far as I was thinking of the mirror architecture I considered this possibility, I also saw the mirror engine using NPM couchdb and it seemed interesting to me as well.

I also considered creating a snapshot mechanism with a compressed file to allow loading the entire structure with a maximum of one week late, with incremental checking the difference would be resolved without any problem.

On the possibility of using git, I quite liked the idea but got caught up in a problem that would be directed to packagist.org, how to keep the git storage down to the main repository? Although the packages are small, the current server has a traffic of approximately 2GB, would it be close to that increment in the git repository, how could a mirror save disk space?

Currently my mirror uses Amazon's smallest server with only 8GB of SSD and sustained use of a maximum of 140MB of RAM. The space used for the server is 1.6GB, with 900MB of gz and 600MB of symbolic link.
With mirror balancing the time to assemble a mirror is 3 to 10 minutes. With updates that last at most 10 seconds and can check for updates using only memory.

One consideration I am still making is to create a complete mirror and leave it inside an image of the docker. So it is possible to update the image every week and allow the creation of the initial mirror without using the infrastructure of other mirrors, but this method has some limitations, such as access to the data.

[webysther]

In making this mirror I considered some important aspects:

• There is a clear interest of the composer's maintainers in not keeping this up, it is justifiable why it will not be the whole community that would want this. This structure needs to depend as little as possible on packagist.org to work, any idea that needs intervention from them will be more difficult.

• The structure needs to be scalable. It is clear that each new public mirror will make it easier to create other mirrors and will reduce downtime in case of re-building.

• It is necessary to use little resource, disk mainly. As growth is exponential I have changed the way nginx works so that compressed json files are a first class citizen. If someone requests the files without compression, the server does this in real time.

• It is necessary to update the mirror as little as possible. That's why the whole structure was thought to have a simple static server, with only two configurations changed and docker. You only need docker and nginx on the server.

[webysther]

I'll create extensive documentation after reviewing the code to reduce cyclomatic complexity and perform the tests. Currently I was able to create the mirror with the smaller AWS server, GCP and Digital Ocean.

[webysther]

I would just like to see packagist.org devote a page to talk about it to anyone who has an interest in the main site, currently I will create a website with their own identity, etc. I would really be happy to see this coming out of their site, even though they make it quite explicit that they give no support to the solution.

The only place that has this information is here:

https://github.com/jakoch/awesome-composer#packagist-mirrors

[ryanaslett]

how could a mirror save disk space?

packagist.org wouldn't need to keep all history forever. probably no more than a days worth. You can clean up a local clone by shallow cloning itself and doing a git garbage collection, so the mirror could be only as large as all of the metadata on disk.

Any solution that relies on http to keep the mirror up to date will inevitibly have to make a request to packagist for each changed file. Git already packages all changes since the last pull into a packfile, reducing the impact of mirrors on packagist itself.

[webysther]

Any solution that relies on http to keep the mirror up to date will inevitibly have to make a request to packagist for each changed file

Not really. Any mirror can use another mirror.

We can use webhook to say packages.json is outdated, but need official support for this feature by team of packagist.

[IndraGunawan]

Indonesia mirror https://packagist.phpindonesia.id/

if you all want to create your own packagist mirror site, you can use this https://github.com/IndraGunawan/packagist-mirror

[webysther]

@IndraGunawan For the next few weeks i put stable and create a tag for my repository and rebuild the view to multiple mirrors around of globe, if you want to join: https://github.com/Webysther/packagist-mirror

[webysther]

The latest version is possible using multiples mirrors and fallback to main mirror if some package metadata don't exist:

# Main mirror used to get providers
MAIN_MIRROR=https://packagist.org

# Data mirror used to download repositories data
DATA_MIRROR=https://packagist.jp,https://packagist.com.br,https://packagist.phpindonesia.id

# Max connections by mirror
MAX_CONNECTIONS=50

[webysther]

Creating a new mirror is pretty fast. :)

[webysther]

Incremental update is smart and fast:

[webysther]

Stable version finished:

https://github.com/Webysther/packagist-mirror
https://packagist.com.br

[webysther]

Thank you @IndraGunawan for the great and simple interface

[IndraGunawan]

https://twitter.com/packagist/status/1012305364614860800

[Seldaek]

Ah yeah thanks @IndraGunawan I forgot to update here. But indeed the situation for Asia should be much better now as we have a server in Singapore, that should at least reduce latency. It's still a bit on the limit in terms of bandwidth at peak time but we'll get more in the next week or two. For South America it's still a problem I guess, if someone knows a good hosting provider doing unlimited bandwidth servers/VPS there let me know and maybe we can get that sorted as well.

[webysther]

In the coming months I will be moving to Europe, I would very much like to have had a more community-based, rather than centralized power response to improving the reusable PHP software distribution. I do not see how the composer / packagist could run with just a small group running as it is running. So I decided (as has already happened with many other OSS) to create a site for the mirrors and if it does not have a good answer to divide the power across the world, create a composer fork or another tool like it happened with NPM and Yarn.

It's sad to see that @Seldaek only responds to this issue when it's a relevant item to. I have a deep admiration for all the work done, but like in Debian, Wikimedia, RHEL, NPM and many others. The possibility of mirroring should be encouraged to third parties and not avoided.

[Seldaek]

@webysther the problem is that embracing community efforts like this introduces security concerns (the repo servers control which code you install/download, that's a big responsibility), and takes a lot of time to manage. We are also looking at changing some of the repository format, so having a lot of third party run sites means we have to coordinate with all of them to avoid breakage, which again makes us spend more time.. It's easy to complain but my time is limited and I try to make the most of it (which is also why I didn't reply before, because I didn't have anything to contribute to this discussion really).

[webysther]

@Seldaek

These problems have already been overcome in different ways in other mirrors of other technologies (debian, npm, etc). As you yourself did a long time ago when you started working on the composer, you were looking for the best solution to solve the problem of recursive depedency.

The only thing I would like was for you to offer a webhook of updates or a session on composer/packagist website to list the mirrors. In addition to having a clear disclamer about how the use of mirrors can affect in uses them.

Changing the structure is not a problem, after defining how it will be, creating an explaining document or section on the composer's website and giving the mirrors time to change their logic, or a new repository that allows third parties to create a mirror with ease. You may feel free to use the various implementations that made it as a template.

If it's not in your best interest, that's fine. Just make it clear so we can solve it. Appearing with the mirror created for a region out of nowhere only shows a lack of alignment with the work we are doing, the history of that issue shows this.

Your previous question: You can create a mirror in Brazil using Azure, GCP or AWS.

[Seldaek]

@webysther I published some details at https://packagist.org/mirrors - I hope this helps go in the direction you meant, obviously improvements are welcome.