Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DI] fix preloading script generation #36103

Merged
merged 1 commit into from Mar 18, 2020
Merged

Conversation

@nicolas-grekas
Copy link
Member

nicolas-grekas commented Mar 16, 2020

Q A
Branch? 4.4
Bug fix? yes
New feature? no
Deprecations? no
Tickets -
License MIT
Doc PR -

(fabbot failure is a false positive)

On master, we should work on being able to preload more classes (esp. all cache-warmup artifacts).

But for 4.4, this is good enough. Submitted as a bug fix because 1. the current code that deals with preloading kinda-works, but only on "dev" mode... and 2. fixing it provides a nice boost!

Small bench on a hello world:

  • before: 380 req/s
  • after: 580 req/s

That's +50%!

Pro-tip: adding a few class_exists() as done in this PR for the classes that are always used in the implementations (e.g. new Foo() in the constructor) will help the preload-script generator to work optimally. Without them, it will discover the symbols to preload only if they're found on methods.

Some of those class_exists() are mandatory, in relation to anonymous classes and https://bugs.php.net/79349

@teohhanhui

This comment has been minimized.

Copy link
Contributor

teohhanhui commented Mar 17, 2020

This pollutes the codebase... 😞

@Pierstoval

This comment has been minimized.

Copy link
Contributor

Pierstoval commented Mar 17, 2020

Is there a way to provide a "default" preloading script for all packages (that will execute these class_exists() calls) and generate a preloading script that would load all available preloading scripts in Symfony packages? This would allow to decouple the preloading from the classes themselves.

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

nicolas-grekas commented Mar 17, 2020

This pollutes the codebase... disappointed

see the description why this is needed.

Is there a way to provide a "default" preloading script for all packages

That's not possible, because the list of classes that need to be preloaded is conditional: vendor/ containing a package doesn't mean in any way it's going to be used at runtime. Putting the class_exists() calls in each files is what makes the list conditional - with the DI container in control of what is actually wired.

@teohhanhui

This comment has been minimized.

Copy link
Contributor

teohhanhui commented Mar 17, 2020

Yeah, I know it's necessary to fix the preload, but is this really worth it?

@teohhanhui

This comment has been minimized.

Copy link
Contributor

teohhanhui commented Mar 17, 2020

What I mean is, even if the preload is not perfect, it's not fundamentally broken, even if the performance is not as good as it could be.

Is getting preload working such a priority that it's worth peppering these all over the codebase?

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

nicolas-grekas commented Mar 17, 2020

@teohhanhui the numbers speak by themselves I think...

Copy link
Member

Nyholm left a comment

Thank you Nicolas for working on this. I like that you managed to add tests to this feature.

I agree that this adds “class_exists” all over the place that can be hard to understand why. But once this PHP feature has matured one can safely remove these extra lines of code.

The performance benefit is impressive. Big 👍 for me.

@rvanlaak

This comment has been minimized.

Copy link
Contributor

rvanlaak commented Mar 17, 2020

The class_exists call does not give many explicit info about why adding these lines is required for the preloading to do it's job.

Would introducing a new trait method to provide additional context via the method's name and phpdoc provide the same performance boost?

trait PreloaderTrait {
    public function preload(string ... $classNames): void
    {
        foreach($classNames as $className) {
            class_exists($className);
        }
    }
}
use Symfony\PreloaderTrait;

preload(
    FilesystemCache::class,
    CoreExtension::class,
    EscaperExtension::class,
    OptimizerExtension::class,
    StagingExtension::class,
    ExtensionSet::class
);
@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

nicolas-grekas commented Mar 17, 2020

But once this PHP feature has matured one can safely remove these extra lines of code.

Actually not: this will remain and should become a new practice: we'll have to tell somewhere which classes are always needed by the implementations.

Here are the options:

  1. in a separate file. For Symfony that'd typically mean in service definitions. Using a tag looks the most sensible, thus this project card.
  2. in the very same file as the implementation using class_exists() (what this PR does)
  3. in the very same file as the implementation, but using a convention (e.g. docblock annotations on the classes, or a special private const OPCACHE_PRELOAD = [TheClasses::class, etc.], or similar.

3. might sound nice to most, but requires coordination in the PHP ecosystem: every project needs to embrace the convention or it will not work.
2. just works everywhere, no need for any special coordination in the ecosystem
1. is a band-aid we're going to need to take over packages that won't do 2. (or 3. if it becomes a thing).

@rvanlaak this doesn't work, we need code that runs - a trait doesn't "run".

@teohhanhui

This comment has been minimized.

Copy link
Contributor

teohhanhui commented Mar 17, 2020

The class_exists option seems the least maintainable, and is arguably also not PSR-1 compliant:

Files SHOULD either declare symbols (classes, functions, constants, etc.) or cause side-effects (e.g. generate output, change .ini settings, etc.) but SHOULD NOT do both.

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

nicolas-grekas commented Mar 17, 2020

The class_exists option seems the most maintainable to me. Same locality, greatest portability.

@ludofleury

This comment has been minimized.

Copy link

ludofleury commented Mar 17, 2020

Impressive performance. at +50% it worth the maintenance cost. Hands down.

@nicolas-grekas nicolas-grekas force-pushed the nicolas-grekas:fix-preload branch from 312a112 to a10fc4d Mar 17, 2020
@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

nicolas-grekas commented Mar 17, 2020

Now with this comment before all calls:
// Help opcache.preload discover always-needed symbols

@Nyholm

This comment has been minimized.

Copy link
Member

Nyholm commented Mar 17, 2020

👍

The comments sure are helpful

@Plopix

This comment has been minimized.

Copy link
Contributor

Plopix commented Mar 18, 2020

+1 for me as well.
+50% !? Almost everything is justified for +50% perfs! And few lines to declare stuff are definitely worth it.

As usual: thanks a lot!

@javiereguiluz

This comment has been minimized.

Copy link
Member

javiereguiluz commented Mar 18, 2020

Thanks Nicolas! This is truly impressive!

Just asking: have you tried these changes in the Symfony Demo app? Should we expect similar ~50% performance improvements? Thanks!

@fabpot
fabpot approved these changes Mar 18, 2020
@fabpot

This comment has been minimized.

Copy link
Member

fabpot commented Mar 18, 2020

Thank you @nicolas-grekas.

@fabpot fabpot merged commit e457b24 into symfony:4.4 Mar 18, 2020
1 of 3 checks passed
1 of 3 checks passed
continuous-integration/travis-ci/pr The Travis CI build failed
Details
fabbot.io Some changes should be done to comply with our standards.
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
@nicolas-grekas nicolas-grekas deleted the nicolas-grekas:fix-preload branch Mar 18, 2020
@fabpot

This comment has been minimized.

Copy link
Member

fabpot commented Mar 18, 2020

This is the same idea as when we add list of classes that we were concatenating in one file to make things faster. Definitely worth the extra manual work.

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

nicolas-grekas commented Mar 18, 2020

Thanks @fabpot

@javiereguiluz no way you'll see 50% on the demo apps. It'd be surprising if it'd spend that much time doing only loading. I hope real apps do a bit more :)

@Nyholm Nyholm mentioned this pull request Mar 21, 2020
2 of 3 tasks complete
This was referenced Mar 27, 2020
fabpot added a commit that referenced this pull request Apr 5, 2020
…lare extra classes to preload/services to not preload (nicolas-grekas)

This PR was merged into the 5.1-dev branch.

Discussion
----------

[DI] add tags `container.preload`/`.no_preload` to declare extra classes to preload/services to not preload

| Q             | A
| ------------- | ---
| Branch?       | master
| Bug fix?      | no
| New feature?  | yes
| Deprecations? | no
| Tickets       | -
| License       | MIT
| Doc PR        | -

To allow fine-grained declaration of sidekick classes in DI extensions.
Follows #36103

Commits
-------

fb04711 [DI] add tags `container.preload`/`.no_preload` to declare extra classes to preload/services to not preload
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

10 participants
You can’t perform that action at this time.