Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

[POC] Optimize performance and resources of ComposerRepository #8275

Closed

Conversation

@Toflar
Copy link
Contributor

Toflar commented Aug 13, 2019

I noticed that Composer always loads all of the package provider data into memory even if it could know exactly that this information is never going to be needed because it cannot resolve to valid set of dependencies.

The Pool already knew about the root requirements (I renamed the variable because filterRequires felt awkward) but not about the root conflicts, so I added that.

What happens now is that if you required for example symfony/framework-bundle: ^4.3, every version that does not match is not loaded into memory anymore, thus speeding up the whole process and reducing memory usage.
The same applies for conflicts.

In case you wonder what happens, if you add a conflict that strips all tags from a package, just try it out by e.g. configuring a conflict to everything lower than symfony/framework-bundle in version 5.0 (which doesn't exist):

{
    "name": "my/project",
    "type": "project",
    "require": {
        "php": "^7.1",
        "symfony/framework-bundle": "^4.3"
    },
    "conflict": {
        "symfony/framework-bundle": "<5"
    }
}

Result:

  Problem 1
    - symfony/framework-bundle v4.3.0 conflicts with my/project[No version set (parsed as 1.0.0)].
    - symfony/framework-bundle v4.3.1 conflicts with my/project[No version set (parsed as 1.0.0)].
    - symfony/framework-bundle v4.3.2 conflicts with my/project[No version set (parsed as 1.0.0)].
    - symfony/framework-bundle v4.3.3 conflicts with my/project[No version set (parsed as 1.0.0)].
    - Installation request for my/project No version set (parsed as 1.0.0) -> satisfiable by my/project[No version set (parsed as 1.0.0)].
    - Installation request for symfony/framework-bundle ^4.3 -> satisfiable by symfony/framework-bundle[v4.3.0, v4.3.1, v4.3.2, v4.3.3].

So as you can see, it clearly states that it "conflicts with my/project", so the root composer.json 馃槉

By adding more conflicts or better requires I can now considerably reduce memory usage and speed up the resolving process.

@nicolas-grekas

This comment has been minimized.

Copy link
Contributor

nicolas-grekas commented Aug 13, 2019

It'd be interesting to have some numbers on real-world apps (flex disabled of course)
Then, I'll let Jordi and Nils be the judge, they're the expert here :)

@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Aug 13, 2019

@alcohol

This comment has been minimized.

Copy link
Member

alcohol commented Aug 14, 2019

For starters, you're missing a return true at the end of isPackageAllowedByRoot

@alcohol

This comment has been minimized.

Copy link
Member

alcohol commented Aug 14, 2019

But even with that, some tests are still failing. A lot less though :-)

@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Aug 14, 2019

Yeah I saw that 馃槃 It was jut a quick hack in the evening with some red wine 馃し鈥嶁檪 馃槃

@Seldaek

This comment has been minimized.

Copy link
Member

Seldaek commented Aug 16, 2019

TBH I'd rather also not do this right now, sounds like a last resort thing rather..

"Forcing" everyone to declare conflicts for all their transitive dependencies sucks. Plus the risk of seeing this practice proliferate all over the place in open source package, would end up having to load many more objects to solve the dependencies than we have now, as we need one conflict object per version of each package, this grows quickly and might even offset the gains I don't know.

@Seldaek

This comment has been minimized.

Copy link
Member

Seldaek commented Aug 16, 2019

To sum up, I have no doubt it'd be effective if well targeted etc, but if you look at the whole ecosystem it sounds like a net negative.

// if the package conflicts with a root conflict, we do not accept the package
if (isset($this->rootConflicts[$package->getName()])) {
$pkgConstraint = new Constraint('==', $package->getVersion());
if ($this->rootConflicts[$package->getName()]->matches($pkgConstraint)) {

This comment has been minimized.

Copy link
@Taluu

Taluu Aug 16, 2019

Contributor

what happens if $ths->rootRequires[$package->getName()] isn't set ? The pckConstraint) won't be set, right ?

@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Aug 19, 2019

The thing is, the solver actually seems to be pretty fast so the ruleset size doesn't seem to be the real problem. It's more the ComposerRepository which just loads a massive amount of JSON data into memory and for some reason lookups slow down (otherwise, why would it be that much faster?).

I understand your point of view saying that it will maybe trigger some bad practice solutions. But fact is, it's already being done. What Symfony Flex does is exactly that and people love it because they just don't care about all the 2.x versions anymore.
I'm just saying that if we introduced that in Composer then Flex could continue to do the same as it does today by just adding conflicts instead of replacing the ComposerRepository class. You cannot have multiple plugins that do the same at the moment. If they added root conflicts, you could.

But I really want to understand why exactly lookups seem to be slowed down that much. I sort of have an idea but I want to look at it more closely before I write reams again :P

@andrerom

This comment has been minimized.

Copy link
Contributor

andrerom commented Aug 19, 2019

Tried this, and did not see any noticeable difference so far.

composer update memory usage on ezsystems/ezpublish-kernel:7.5
PHP: PHP 7.3.6 ( NTS ), Zend OPcache v7.3.6, Xdebug v2.7.2

v1.9.0 --no-plugins:
[347.4MiB/14.44s] Memory usage: 347.4MiB (peak: 1619.91MiB), time: 14.44s

v1.9.0 with symfony flex (+ "extra.symfony.require: 3.4.x"):
[212.5MiB/3.26s] Memory usage: 212.49MiB (peak: 480.82MiB), time: 3.26s

With filter-root-conflicts-and-requires:
[347.5MiB/14.15s] Memory usage: 347.48MiB (peak: 1620.07MiB), time: 14.15s

With filter-root-conflicts-and-requires and conflict rules to rule out Symfony <3.4, >=3.5:
[346.7MiB/14.33s] Memory usage: 346.66MiB (peak: 1619.93MiB), time: 14.33s

Composer 2.0, NB! conflicts on "ocramius/package-versions * requires composer-plugin-api ^1.0", so not an apples to apples comparison:
[108.2MiB/12.96s] Memory usage: 108.2MiB (peak: 1374.8MiB), time: 12.96s 

@Toflar Feel free to message me on twitter to go over testing as I might have overlooked a way to trigger this. But from a user view I have to agree with Jordi that having to define conflicts to rule out versions seems to not be very user friendly, it'll produce a lot of dependency issues once lib maintainers start using it.

@Seldaek That said I hope you agree this is a major/growing issue.
(FYI On PHP 5.6 we have x3, so around 4GiB, which we still need to maintain. And AFAICS within next 2 years numbers will probably grow 2-3 times across PHP versions, as size of package dependencies versions graph exponentially grows)

EDIT: After some talks with @Toflar I'm able to reproduce and even get [197.5MiB/3.17s] Memory usage: 197.52MiB (peak: 230.16MiB), time: 3.17s if I go as far as setting conflicts on all symfony and doctrine package branches I don't care for. But I haven't yet had time to re-test everything and write up how the tests was done for others to reproduce.

Toflar added 2 commits Aug 20, 2019
@Toflar Toflar force-pushed the Toflar:filter-root-conflicts-and-requires branch from cc29ac1 to 4e5f94f Aug 20, 2019
@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Aug 20, 2019

I've adjusted the implementation and here are some numbers. I've tested using the following composer.json:

{
    "name": "my/contao-project",
    "type": "project",
    "require": {
        "php": "^5.6 || ^7.0",
        "contao/calendar-bundle": "^4.4",
        "contao/comments-bundle": "^4.4",
        "contao/faq-bundle": "^4.4",
        "contao/listing-bundle": "^4.4",
        "contao/manager-bundle": "4.4.*",
        "contao/news-bundle": "^4.4",
        "contao/newsletter-bundle": "^4.4"
    }
}

The command I run is bin/composer update --dry-run --profile.

  • master@bfba228b5a232bcb9e4bb7941f0a0aaa37bab117
    Memory usage: 366.82MiB (peak: 1875.88MiB), time: 28.75s

  • with this PR
    Memory usage: 365.03MiB (peak: 1873.79MiB), time: 25.21s

So there's an ever so slight improvement because of the root requirements filtered out. However, there are not too many tags in that particular case so gains are marginal.
Now let's add some conflicts:

{
    "name": "my/contao-project",
    "type": "project",
    "require": {
        "php": "^5.6 || ^7.0",
        "contao/calendar-bundle": "^4.4",
        "contao/comments-bundle": "^4.4",
        "contao/faq-bundle": "^4.4",
        "contao/listing-bundle": "^4.4",
        "contao/manager-bundle": "4.4.*",
        "contao/news-bundle": "^4.4",
        "contao/newsletter-bundle": "^4.4"
    },
    "conflict": {
        "symfony/console": "<3.3",
        "symfony/filesystem": "<3.3",
        "symfony/framework-bundle": "<3.3",
        "symfony/http-foundation": "<3.3",
        "symfony/security": "<3.3",
        "symfony/yaml": "<3.3",
        "symfony/event-dispatcher": "<3.3",
        "symfony/options-resolver": "<3.3",
        "symfony/http-kernel": "<3.3",
        "doctrine/dbal": "<2.5"
    }
}
  • master@bfba228b5a232bcb9e4bb7941f0a0aaa37bab117
    Memory usage: 366.82MiB (peak: 1876.58MiB), time: 27.98s
  • with this PR
    Memory usage: 312.78MiB (peak: 1092.01MiB), time: 14.68s

So it is considerably faster and more memory efficient 馃槃
Of course you could also move the conflicts to the requirements. It would work just the same and have the same effect (I did test that).

I've also updated the problem information. So if I conflict with my own requirements like so:

{
    "name": "my/contao-project",
    "type": "project",
    "require": {
        "php": "^5.6 || ^7.0",
        "contao/calendar-bundle": "^4.4",
        "contao/comments-bundle": "^4.4",
        "contao/faq-bundle": "^4.4",
        "contao/listing-bundle": "^4.4",
        "contao/manager-bundle": "4.4.*",
        "contao/news-bundle": "^4.4",
        "contao/newsletter-bundle": "^4.4"
    },
    "conflict": {
        "contao/manager-bundle": "^4.4"
    }
}

I get this:

Your requirements could not be resolved to an installable set of packages.
  Problem 1
    - The requested package contao/manager-bundle 4.4.* was excluded by your own root composer.json "conflict" definition.

It also works for conflicts of transitive dependencies:

{
    "name": "my/contao-project",
    "type": "project",
    "require": {
        "php": "^5.6 || ^7.0",
        "contao/calendar-bundle": "^4.4",
        "contao/comments-bundle": "^4.4",
        "contao/faq-bundle": "^4.4",
        "contao/listing-bundle": "^4.4",
        "contao/manager-bundle": "4.4.*",
        "contao/news-bundle": "^4.4",
        "contao/newsletter-bundle": "^4.4"
    },
    "conflict": {
        "doctrine/dbal": ">=2.5"
    }
}

This is what you get:

[...]
    - contao/core-bundle 4.4.1 requires doctrine/dbal ^2.5 -> conflicts with your root composer.json "conflict" definition.
    - contao/core-bundle 4.4.0 requires doctrine/dbal ^2.5 -> conflicts with your root composer.json "conflict" definition.
    - Installation request for contao/manager-bundle 4.4.* -> satisfiable by contao/manager-bundle[4.4.0, 4.4.1, 4.4.10, 4.4.11, 4.4.12, 4.4.13, 4.4.14, 4.4.15, 4.4.16, 4.4.17, 4.4.18, 4.4.19, 4.4.2, 4.4.20, 4.4.21, 4.4.22, 4.4.23, 4.4.24, 4.4.25, 4.4.26, 4.4.27, 4.4.28, 4.4.29, 4.4.3, 4.4.30, 4.4.31, 4.4.32, 4.4.33, 4.4.34, 4.4.35, 4.4.36, 4.4.37, 4.4.38, 4.4.39, 4.4.4, 4.4.40, 4.4.41, 4.4.42, 4.4.5, 4.4.6, 4.4.7, 4.4.8, 4.4.9].
@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Aug 20, 2019

And here is a comparison for a typical Symfony application (without Flex) plus all the requirements of api-pack so API platform dependencies.
Note that I did not specify any special conflicts! The optimizations kick in by just having the root requirements filter out impossible candidates in that case:

{
    "name": "my/symfony-project",
    "type": "project",
    "require": {
        "php": "^7.2.9",
        "ext-ctype": "*",
        "ext-iconv": "*",
        "symfony/console": "^4.3",
        "symfony/dotenv": "^4.3",
        "symfony/framework-bundle": "^4.3",
        "symfony/yaml": "^4.3",

        "api-platform/core": "^2.1",
        "doctrine/annotations": "^1.0",
        "doctrine/orm": "^2.4.5",
        "doctrine/doctrine-bundle": "^1.6",
        "nelmio/cors-bundle": "^1.5",
        "phpdocumentor/reflection-docblock": "^3.0 || ^4.0",
        "symfony/asset": "^4.3",
        "symfony/expression-language": "^4.3",
        "symfony/security-bundle": "^4.3",
        "symfony/twig-bundle": "^4.3",
        "symfony/validator": "^4.3"
    }
}
  • master@bfba228b5a232bcb9e4bb7941f0a0aaa37bab117
    Memory usage: 273.58MiB (peak: 661.23MiB), time: 7.34s

  • 2.0@4dabc17ec1e55d42d9611271d1728dcf9f79d56a
    Memory usage: 43.51MiB (peak: 448.84MiB), time: 7.02s

  • with this PR
    Memory usage: 247.97MiB (peak: 635.68MiB), time: 6.57s

I mean, it's not huge but it's 10% less memory used anyway compared to current master, without finetuning anything.
And now if I were to analyze what packages it installs, I can optimize a bit more by adding a few conflicts (I don't need to specify thousands of packages, it kicks in pretty quickly):

{
    "name": "my/symfony-project",
    "type": "project",
    "require": {
        "php": "^7.2.9",
        "ext-ctype": "*",
        "ext-iconv": "*",
        "symfony/console": "^4.3",
        "symfony/dotenv": "^4.3",
        "symfony/framework-bundle": "^4.3",
        "symfony/yaml": "^4.3",

        "api-platform/core": "^2.1",
        "doctrine/annotations": "^1.0",
        "doctrine/orm": "^2.4.5",
        "doctrine/doctrine-bundle": "^1.6",
        "nelmio/cors-bundle": "^1.5",
        "phpdocumentor/reflection-docblock": "^3.0 || ^4.0",
        "symfony/asset": "^4.3",
        "symfony/expression-language": "^4.3",
        "symfony/security-bundle": "^4.3",
        "symfony/twig-bundle": "^4.3",
        "symfony/validator": "^4.3"
    },
    "conflict": {
        "symfony/yaml": "<4.3",
        "symfony/serializer": "<4.3",
        "symfony/http-foundation": "<4.3",
        "symfony/filesystem": "<4.3",
        "symfony/cache": "<4.3",
        "symfony/config": "<4.3"
    }
}
  • with this PR
    Memory usage: 236.28MiB (peak: 569.22MiB), time: 5.95s

And now I can go a bit crazy and add some more conflicts (or even better, only add them to the composer.json dynamically during the CI job) because my CI does not offer 570 MB of RAM:

{
    "name": "my/symfony-project",
    "type": "project",
    "require": {
        "php": "^7.2.9",
        "ext-ctype": "*",
        "ext-iconv": "*",
        "symfony/console": "^4.3",
        "symfony/dotenv": "^4.3",
        "symfony/framework-bundle": "^4.3",
        "symfony/yaml": "^4.3",

        "api-platform/core": "^2.1",
        "doctrine/annotations": "^1.0",
        "doctrine/orm": "^2.4.5",
        "doctrine/doctrine-bundle": "^1.6",
        "nelmio/cors-bundle": "^1.5",
        "phpdocumentor/reflection-docblock": "^3.0 || ^4.0",
        "symfony/asset": "^4.3",
        "symfony/expression-language": "^4.3",
        "symfony/security-bundle": "^4.3",
        "symfony/twig-bundle": "^4.3",
        "symfony/validator": "^4.3"
    },
    "conflict": {
        "symfony/yaml": "<4.3",
        "symfony/serializer": "<4.3",
        "symfony/http-foundation": "<4.3",
        "symfony/filesystem": "<4.3",
        "symfony/cache": "<4.3",
        "symfony/config": "<4.3",
        "symfony/routing": "<4.3",
        "symfony/finder": "<4.3",
        "symfony/var-exporter": "<4.3",
        "symfony/expression-language": "<4.3",
        "symfony/security-http": "<4.3",
        "symfony/security-guard": "<4.3",
        "symfony/security-csrf": "<4.3",
        "symfony/security-core": "<4.3",
        "symfony/dependency-injection": "<4.3",
        "symfony/event-dispatcher": "<4.3",
        "symfony/http-kernel": "<4.3",
        "symfony/debug": "<4.3",
        "symfony/property-access": "<4.3",
        "symfony/property-info": "<4.3",
        "doctrine/dbal": "<2.9",
        "api-platform/core": "<2.4"
    }
}

And voil脿:

  • with this PR
    Memory usage: 193.33MiB (peak: 245.72MiB), time: 4.61s

I'm not saying that Composer should mention this anywhere as being best practice by all means! And I would also prefer to have a solution that works even better out of the box in the long term. But that's going to take some time and with this I could at least get my CI jobs running again on Travis. Because let's face it, the numbers here are still kind of okay but I'm having projects with 3-4 GB memory usage. Right now I'm just lost 馃し鈥嶁檪

@stof

This comment has been minimized.

Copy link
Contributor

stof commented Aug 20, 2019

there is 2 changes here:

  • taking root conflicts into account
  • taking root requirements into account in the ComposerRepository

Your first case with the Symfony API project gets optimized by the second change (which is similar to what flex does, except that Flex does it for all Symfony components, rather than doing it for root requirements). I think this change would be fine (the main argument of @Seldaek against this PR is only against the usage of conflicts, not against the usage of root requirements in more places).

@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Aug 21, 2019

Sure, I understand that there's two optimizations here but I don't understand why it would matter. Filtering out root conflicts is just as good as filtering out on root requirements. Both can and should be done.
The statement that it could lead to an anti-pattern where people just randomly add root conflicts to optimize memory usage doesn't make sense to me because they could also just randomly add root requirements? I fail to see why one should be okay and the other one not.

I think we should do both. Whether or not people are abusing it, is not really something we can and should control.

@andrerom

This comment has been minimized.

Copy link
Contributor

andrerom commented Aug 21, 2019

The statement that it could lead to an anti-pattern where people just randomly add root conflicts to optimize memory usage doesn't make sense to me because they could also just randomly add root requirements? I fail to see why one should be okay and the other one not.

I agree, and as far as I can read it it seems @stof does too.

But I also agree with @Seldaek that we should try to avoid this becoming a thing (overusing conflicts to workaround memory issues). Afaik the only way to do that is to somehow make sure we go further after this PR to get rid of having to download (in-the-end) unrelated transitive version dependency data, or at least most of it.

@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Aug 21, 2019

Sure, I fully agree with that as well. But here's the thing:

  1. Does it make sense to filter out packages excluded by root requirements? Yes.
  2. Does it make sense to filter out packages excluded by root conflicts? Yes.
  3. Does it make sense to optimize resource usage by using either of them? No.

This PR provides 1) and 2). What the people do with 3) is their own business.
There are loads of things you can do wrong with Composer. Like requiring packages with * or without upper bounds etc. Just because you can doesn't mean it's the right thing to do in general :)

@andrerom

This comment has been minimized.

Copy link
Contributor

andrerom commented Aug 21, 2019

Agreed, to be clear I'm full +1 on merging this to 1.9 馃憤

I can reproduce your gains, in my case I get down from 1600MiB to 230MiB peak memory use if I want to with this. Updated my originally comment to reflect that btw.

What I'm saying is that further work is needed on topic of memory use, probably for 2.0, but that is offtopic on this PR so please ignore that here.

@nicolas-grekas

This comment has been minimized.

Copy link
Contributor

nicolas-grekas commented Aug 21, 2019

I think neither 1. nor 2. make sense, because they are micro-optimizations with no practical benefit.

The numbers above prove it and this matches the theory: a 10% gain is not solving anything when the issue is an order of magnitude away.

What I mean is that the issue happens because the matrix of versions grows in a quadratic way at best. Reducing the list of tags for a few packages doesn't change anything on this level.

Flex succeeds here because it forces Composer to have a narrow view of history, which is the only way to solve the combinatorial explosion. No matter how Composer v1 or v2 is optimized, having a quadratic matrix will always explode.

There are only two ways to solve this: either remove the older tags to keep the matrix in a reasonable size or reduce the big-O magnitude of the algorithm that computes the matrix, if possible at all.

Thanks for digging anyway!
What about trying to implement what I suggested with a PR on https://github.com/composer/packagist?

@nicolas-grekas

This comment has been minimized.

Copy link
Contributor

nicolas-grekas commented Aug 21, 2019

Note that we don't need packagist: someone could very well create a plugin and invite package authors to send PR on it to blacklist abandoned tags, and the plugin would ensure these are excluded as flex does.

It'd prefer having this native to the packagist infra thought :)

@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Aug 21, 2019

The numbers above prove it and this matches the theory: a 10% gain is not solving anything when the issue is an order of magnitude away.

I'm a bit disappointed with this statment. It's still 10%. It's a huge gain! I've seen PR's being merged into Composer for a lot less 馃槃

What about trying to implement what I suggested with a PR on https://github.com/composer/packagist?

I'm hesitant. Not sure what the exact plans for v2 are here.

@nicolas-grekas

This comment has been minimized.

Copy link
Contributor

nicolas-grekas commented Aug 21, 2019

I'm a bit disappointed with this statment. It's still 10%. It's a huge gain! I've seen PR's being merged into Composer for a lot less smile

I didn't write the gain was insignificant. What I mean is that we don't need a linear solution to a quadratic problem. We need a quadratic solution instead, otherwise, we missed solving the issue at all. PRs that have been merged before were linear.

@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Aug 21, 2019

Sure, we need even more improvements, that's for sure :)

@Seldaek

This comment has been minimized.

Copy link
Member

Seldaek commented Aug 21, 2019

@Toflar could you try these benchmarks against the 2.0 branch as well please? Because that should be the baseline for comparison really, even though it's not complete.

@Seldaek

This comment has been minimized.

Copy link
Member

Seldaek commented Aug 21, 2019

Mostly interested in the one without you adding tons of conflicts, because of course that will help but it's also not a real improvement if we require user action.

@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Aug 21, 2019

Sure! I've updated #8275 (comment). Way better than master but of course compared to the one where we filter out more packages, there's still a long way to go 馃槉

The issue here is just really the sheer amount of CompletePackage instances that are being generated and loaded into memory even though they're never used. At least for master that's the case, I haven't checked 2.0 really.

@Seldaek

This comment has been minimized.

Copy link
Member

Seldaek commented Aug 21, 2019

Ok but 2.0 is still better than this PR without composer.json change, which makes me think we should focus on shipping 2.0 instead of discussing whether this PR makes sense.. 2.0 also provides a better base to work on such optimizations, and with COMPOSER_DISABLE_NETWORK=1 (after a first run to prime the cache) you can also better compare run times there. Anyway not saying this PR is dead, just needed to get the rant out I guess.

As for the memory usage, are you sure it's the CompletePackage instances? Last time I checked I think it was more the amount of Rule objects which was the main issue. Obviously that is connected to the amount of packages which are loaded.

@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Aug 21, 2019

Anyway not saying this PR is dead, just needed to get the rant out I guess.

Let's call it what it is already :P I'm trying to find solutions to the problem in the 2.0 branch then. Given the amount of open issues assigned to the 2.0 milestone I just figured it's still going to take a long time and I wanted to have some gains now :)

As for the memory usage, are you sure it's the CompletePackage instances? Last time I checked I think it was more the amount of Rule objects which was the main issue. Obviously that is connected to the amount of packages which are loaded.

I'm not 100% sure but the way it works on master is that the RuleSetGenerator starts with a set of packages (your root requirements) and then asks the Pool for possible candidates (whatProvides()). This again asks all the provider repos (so ComposerRepository) for all the packages but without any special constraint. So here we load a huge amount of nonsense package instances that are internally (in the Pool) kept as reference somehow (Pool::packageById()). They are only filtered out according to the constraint later.

@Seldaek

This comment has been minimized.

Copy link
Member

Seldaek commented Aug 21, 2019

Yes the mechanism you describe was fixed in 2.0 already I believe, but then the gazillion Rule objects still get created when you have tons of versions to look at.

@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Aug 21, 2019

I noticed that things have been moved to Pool. Okay, I'll fiddle around with 2.0 next time then 馃槃

@ausi ausi mentioned this pull request Aug 27, 2019
0 of 5 tasks complete
@Seldaek Seldaek added this to the 2.0-core milestone Aug 30, 2019
@Seldaek

This comment has been minimized.

Copy link
Member

Seldaek commented Jan 14, 2020

Sorry but trying to clean things up, @Toflar is this still relevant for the 2.0 branch you think? If so it needs to be rebased.

@Seldaek

This comment has been minimized.

Copy link
Member

Seldaek commented Jan 14, 2020

I believe it might have been done by #7631 but haven't looked at this PR again in details.

@Seldaek

This comment has been minimized.

Copy link
Member

Seldaek commented Jan 14, 2020

Actually I guess it can be closed and replaced by #8295 - sorry for the spam

@Toflar

This comment has been minimized.

Copy link
Contributor Author

Toflar commented Jan 14, 2020

We can close this PR because it cannot be merged as is.
Basically this is the foundation for #8295. They both kind of have to be combined. #8295 misses parts that are already implemented here.
But both would need to be adjusted to the new PoolBuilder so I guess we'll end up having a completely new PR.

@Seldaek Seldaek closed this Jan 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can鈥檛 perform that action at this time.