New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Properly handle optimized .pyc files in loader in PY3 #48636

Merged
merged 14 commits into from Aug 8, 2018

Conversation

Projects
None yet
7 participants
@terminalmage
Member

terminalmage commented Jul 17, 2018

See #48195 (review) for some background.

This PR properly handles optimized .pyc files in the loader. It also introduces a new config option, optimization_order, which allows for the priority to be set so that optimized .pyc files are preferred.

It also fixes issue #46924 (separately fixed in #48593) in which the loader preferred .pyc files over .py files of the same name in Python 3, causing updated .py files not to be loaded. Note that this means that optimization_order only comes into play when Salt is distributed without .py files. NOTE: since #48593 was merged, my commit that made the same fix was a no-op and has disappeared in my latest rebase.

Unit tests added to confirm that the optimization_order option works as expected.

In addition to the above, this PR makes the following two changes:

  1. Because we do an os.listdir() on each module directory processed by the loader, this means that the __pycache__ dir was in the results. It would never end up being loaded, so this didn't break anything, but I have nonetheless excluded it from the os.listdir() results so that the loader doesn't waste any effort processing it.

  2. The suffix map/order are now compiled in the LazyLoader's dunder init. The values which are used to build both are module-level globals, so there is no need to repeat this every time the file mapping is refreshed.

@cachedout due to the fact that the optimization_order only matters when there is no .py file for that module, I decided not to add release notes for this. It is the corner-iest of corner cases.

@terminalmage terminalmage requested a review from saltstack/team-core as a code owner Jul 17, 2018

@salt-jenkins salt-jenkins requested a review from saltstack/team-suse Jul 17, 2018

@terminalmage

This comment has been minimized.

Member

terminalmage commented Jul 17, 2018

The tests I added need some work, I'm getting test failures when I run the unit test locally on Cent 7 (which has Python 3.4 and is therefore using imp instead of importlib).

@terminalmage terminalmage force-pushed the terminalmage:loader-fixes branch from cc6b5e7 to 1fdc318 Jul 17, 2018

@terminalmage

This comment has been minimized.

Member

terminalmage commented Jul 17, 2018

OK, tests should work now. Also added a new one.

@ari

This comment has been minimized.

Contributor

ari commented Jul 18, 2018

Can you explain the point of optimization_order? Why would a user ever wish to set this value away from the default?

@terminalmage

This comment has been minimized.

Member

terminalmage commented Jul 18, 2018

If one wanted to get Salt to load anything but the default (unoptimized) .pyc file, they would need a way to tell Salt to do so. That's what this option is for.

@ari

This comment has been minimized.

Contributor

ari commented Jul 18, 2018

So on a platform that creates the optimised files (eg FreeBSD) by default, they will be ignored and the .pyc used instead? And if the .py source file is newer than the .pyc compiled file, then the source file will be used?

Is it too resource draining to set default optimization_order so that files are loaded from most optimised to least (opt-2, opt-2, pyc, py) unless the py file is newer that the others?

I haven't done any benchmarking, but anything to make salt go faster is welcome.

@terminalmage

This comment has been minimized.

Member

terminalmage commented Jul 18, 2018

As noted in the OP, the optimization_order value only comes into play when Salt is distributed without .py files for its modules.

@ari

This comment has been minimized.

Contributor

ari commented Jul 18, 2018

Sorry to be dense. If there are source files, will this now mean compiled files are ignored or used in order from most optimised?

@isbm

isbm approved these changes Jul 19, 2018

@rallytime

This comment has been minimized.

@terminalmage

This comment has been minimized.

Member

terminalmage commented Jul 23, 2018

@ari If there are source files, it means that .pyc files will not be loaded. The normal means of loading source files will be used (which will also update the .pyc if needed).

Until #48593 was merged a couple days ago, Python 3 would prefer .pyc files of the same module name, but this would have the effect not picking up changes to the .py file if it were updated.

@terminalmage

This comment has been minimized.

Member

terminalmage commented Jul 23, 2018

I'm working on the lint failures now, as well as another Python 3-related loader issue.

@ari

This comment has been minimized.

Contributor

ari commented Jul 23, 2018

@terminalmage OK, so then if we want maximum performance from salt, we need to remove all the source files from the deployment and leave behind only the opt-1 or opt-2 optimised compiled files? Is there no way to have salt automatically load the most optimised files it can find (as long as they are the same age as the .py files)?

@terminalmage

This comment has been minimized.

Member

terminalmage commented Aug 6, 2018

@ari No, there is no way to do that right now, and I'm not sure it would be a good idea to do that since stat'ing every module every time the loader is refreshed would consume additional resources, and I don't know that the net improvement gain by loading .pyc files would be worth it.

@terminalmage terminalmage force-pushed the terminalmage:loader-fixes branch from 3f06c27 to dd06479 Aug 6, 2018

@rallytime

This comment has been minimized.

Contributor

rallytime commented Aug 7, 2018

@terminalmage Something isn't quite right here with the Py3 tests. Can you take another swing at this?

terminalmage added some commits Jul 13, 2018

PY3: Support different optimization levels
This uses the optimization_order config option to control which module
is loaded when multiple optimization levels have been installed.
Don't put __pycache__ dir in the file list
This prevents the loader from needlessly trying to process it (which
will fail anyway since it's not a file).
Only compile the suffix_order/map once per LazyLoader instance
The values that dictate this are module-level globals, so repeating this
every time we refresh the file mapping is unnecessary.

terminalmage added some commits Jul 17, 2018

Add a test to confirm that .py files are still loaded correctly
Also, use USE_IMPORTLIB to gate testing on optimization_order, since
this only works with Python 3.5+.

@terminalmage terminalmage force-pushed the terminalmage:loader-fixes branch from cbb2376 to 0441cd5 Aug 8, 2018

@rallytime rallytime merged commit 41bd368 into saltstack:2017.7 Aug 8, 2018

8 checks passed

WIP ready for review
Details
continuous-integration/jenkins/pr-merge This commit looks good
Details
jenkins/pr/docs The docs job has passed
Details
jenkins/pr/lint The lint job has passed
Details
jenkins/pr/py2-centos-7 The py2-centos-7 job has passed
Details
jenkins/pr/py2-ubuntu-1604 The py2-ubuntu-1604 job has passed
Details
jenkins/pr/py3-centos-7 The py3-centos-7 job has passed
Details
jenkins/pr/py3-ubuntu-1604 The py3-ubuntu-1604 job has passed
Details

@terminalmage terminalmage deleted the terminalmage:loader-fixes branch Aug 8, 2018

@rhavenn

This comment has been minimized.

rhavenn commented Aug 15, 2018

Will this be included in a 2018.3.3 release? Is there an ETA of said release?

FreeBSD and Python 3.6 user here and salt is more or less broken at the moment for me.

@gtmanfred

This comment has been minimized.

Contributor

gtmanfred commented Aug 15, 2018

yes, we merge forward to 2018.3 before branching the 2018.3.3 release branch.

the target is by the end of this month.

@cedwards

This comment has been minimized.

Member

cedwards commented Aug 22, 2018

@rhavenn I will be testing this patch in the FreeBSD port tonight. Fingers crossed we can patch this mid-stream in FreeBSD until it is available in 2018.3.3.

@rhavenn

This comment has been minimized.

rhavenn commented Aug 22, 2018

@cedwards Sweet!!! Thanks.

@cedwards

This comment has been minimized.

Member

cedwards commented Aug 30, 2018

I've been working on this, but I'm still having trouble. Need more time to troubleshoot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment