make use of packages.json includes to make hash based caching possible

[Flyingmana]

No description provided.

[Flyingmana]

refactoring some parts completely would probably make more sense, but this way we have at least a simple caching, which is for some users noticeable when the packages.json is already over 10M big.

[stof]

@Seldaek should we build it on the providers system rather than the old includes system ?

[naderman]

actually I think we should get rid of the providers eventually as we don't need that mechanism anymore after the changes to how provide/replace work. We know an exact list of package names now.

[naderman]

So using the old includes system here is actually fine I think, and we'll need to come up with something new to make things more efficient in the long run.

[Flyingmana]

Yeah, for the actual issue of caching, this is the simplest solution.
the providers system is more for splitting them complete by package name, which on the downside could create a lot of requests on the first usage. For our use case there were some possibilities to optimize, but I dont have an idea to make them reusable for others. Also @Seldaek mentioned he works on a alternative, so I suggest we wait for this, before we invest to much time.

[stof]

@naderman the providers system in the repository spec is still needed: it allows loading only some of the subfiles. The old includes on the other hand simply splits into mutliple files without structure (the first version of packagist used this to group old releases together as the file was expected to change less often). Using includes only in the repository spec would mean you load the 103 914 packages available on Packagist each time you run composer update, and you will then need several GB of memory for this.

For small Satis repos, includes will generally be enough indeed, but then, some people are also using Satis with bigger repos (mirroring part of Packagist for instance). So the question of using one or the other still stands.
Using includes here improves caching indeed, but it still loads all metadata at once, thus not helping for bigger repos

[naderman]

@stof I am aware of what the point of providers is. I was saying we will need something else, as we will want to load individual packages not everything providing a name anymore. Please read my response. Hence includes here will be just fine for now, until we come up with a new mechanism instead of providers which we can then also use here. Which is what I said to begin with.

[Flyingmana]

so is there anything blocking the merge of this pull request I can help with?

[naderman]

Nope, all good.