WIP: add options to preserve environment variables #4401

reorx · 2019-08-28T12:24:52Z

add arg --pyiboot01-preserved-envs with default value VIRTUAL_ENV

An additional pyiboot01_bootstrap.py file will be generated to workpath
during build process, which contains the modified version of the source.
This mechanism provides fine-granular control over what pyiboot01 does
when running the app, more options (template vars) could be added in the future
if similar feature is requested.
add arg --pyiboot01-no-preserved-envs to control --pyiboot01-preserved-envs availability

If the user does not want any env vars to be preserved, passing this argument will simply
turn off this feature

even though we have enough reason to delete VIRTUAL_ENV in environment, we should still keep the original data for those who relies on it essentially, provide another way to access it when necessary.

PyInstaller/loader/pyiboot01_bootstrap.py

reorx · 2019-08-30T09:31:09Z

The newest commit contains the more generic solution about environment preserving, please check.

I'm still trying to figure out how to add tests about this part, will update when I finish that.

htgoebel

I now found time to have a deeper look into the code. See my inline comments.

Anyway: This pull-request should either be "add renamed env var to replace VIRTUAL_ENV" or something like "Add option to preserve some environment variables."

Please mind to submit a changelog entry so our users can learn about your change. And this is why this pull-request should be either "add renamed env var …" or "Add option to preserve …", since the news-fragment can only ba about one change.

PyInstaller/loader/pyiboot01_bootstrap.py

- add arg --pyiboot01-preserved-envs with default value VIRTUAL_ENV An additional pyiboot01_bootstrap.py file will be generated to `workpath` during build process, which contains the modified version of the source. This mechanism provides fine granular control over what pyiboot01 does when running the app, more options (template vars) could be added in the future if similar feature is requested. - add arg --pyiboot01-no-preserved-envs to control --pyiboot01-preserved-envs availability If the user does not want any env vars to be preserved, passing this argument will simply turn off this feature

htgoebel

Thanks for the update. I added more inline comments.

Of course we also need

documentation - IMHO this option is worth an explanation section in the manual (I suggest doc/runtime-information.rst)
a test-case
The test-case should contain two test_source() calls, like this

# verifying the command line option actually works
pyi_builder.test_source(
"""
import os
assert os.genenv("PRESERVED…")
""",
pyi_args=['--preserve-…])
)
# verify the command-line option is not leaking env-vars to other builds
pyi_builder.test_source(
"""
import os
assert not os.genenv("PRESERVED…")
""")

PyInstaller/building/api.py

htgoebel · 2019-08-30T10:11:33Z

PyInstaller/building/makespec.py

+                   metavar="ENVVAR", dest='pyiboot01_preserved_envs',
+                   help='Environment variables that will be preserved in pyiboot01_bootstrap module, '
+                   'the preserved env var will be renamed with prefix `PYINSTALLER_PRESERVED_`.'
+                   'This option can be used multiple times. (default: VIRTUAL_ENV)')


Option should be just --preserve-envvar. We must not bother users with pyiboot01, which is an implementation detail. Also "pyiboot01_bootstrap module" should not occur in the help-message.

Also this should be singular, since the option adds a single variable to the list.

I'd not add a default here. Most applications do not even VIRTUAL_ENV. Also be can omit the --no-… option.

"Environment variable" (singular!) "to be preserved when running the application".

"preserve" might not be the proper verb at all. I'm not a native speaker, but "retain", "copy", "keep" might be better.

I finally decided to call it --disguise-envvar, as a verb-object phrase, I found the reason why I want to use the word "preserve" is because I only thought about the result of the env vars, they are indeed preserved, but our purpose is to hide them, or disguise, which has the meaning of change things to a different form, fits better in this situation.

Here's the LDOCE thesaurus of the two words:

hide: to make something difficult to see or find, or to not show your true feelings

disguise: to make someone or something seem like a different person or thing, so that other people cannot recognize them

I'm not a native speaker either, I agree we should choose the word very cautiously, glad to change it to other word if we can find out anything better.

I'd not add a default here. Most applications do not even VIRTUAL_ENV. Also be can omit the --no-… option.

I actually don't want to add the default value either, but I thought it might be friendly to not change the default behavior while adding new features, if you think it's OK, then I'd be happy to remove it and the --no... option (they were already removed in the latest code!).

PyInstaller/depend/analysis.py

PyInstaller/loader/pyiboot01_bootstrap.py

reorx · 2019-08-30T10:39:59Z

Please mind to submit a changelog entry so our users can learn about your change. And this is why this pull-request should be either "add renamed env var …" or "Add option to preserve …", since the news-fragment can only ba about one change.

Just added the news file and updated the title of this PR, forgive me for not reading the contribution guide before pushing the code, got too hurry in a time-limited situation 😢

htgoebel · 2019-08-30T10:40:35Z

PyInstaller/depend/analysis.py

+    # type:(str, dict) -> str
+    for k, v in context.items():
+        content = re.sub(r"'\{\{ " + k + r" \}\}'", repr(v), content)
+    return content


I suggest re.sub(re.escape("'{{%s}}'" % k), …, which is easier to read

Another idea: What about not modifing a copy of pyiboot01_bootstrap, but to generate a new module e.g. pyimod04_variables, which currently only defines PRESERVED_ENVS?

I suggest re.sub(re.escape("'{{%s}}'" % k), …, which is easier to read

Update as suggested

What about not modifing a copy of pyiboot01_bootstrap, but to generate a new module e.g. pyimod04_variables, which currently only defines PRESERVED_ENVS?

Considering that a new file adds more overhead, and a generated new file is more or less the same thing with a modified existing file (which means we won't do anything less in the code), I think using pyiboot01_bootstrap is enough. Also manipulating environment is as dangerous and important as updating sys module, they should sit together as company :D

htgoebel · 2019-08-30T10:56:30Z

Looks like we are working on this the same tim :-) I'll stop reviewing for new, waiting for your code to be settled a bit :-)

also rename the dest arg name pyiboot01_no_preserved_envs to disguised_envs

new function is called `build_from_template`. other changes: - use re.escape for better readability - add unit test for build_from_template_file

reorx · 2019-08-31T10:03:36Z

The code has been updated for all the requests in the review, see the latest 4 commits and replies in the conversations.

reorx · 2019-10-31T12:08:52Z

Almost forget this one, shall we pick it up and start over?

loader.pyiboot01: add renamed env var to replace VIRTUAL_ENV

9a77baa

even though we have enough reason to delete VIRTUAL_ENV in environment, we should still keep the original data for those who relies on it essentially, provide another way to access it when necessary.

htgoebel reviewed Aug 29, 2019

View reviewed changes

PyInstaller/loader/pyiboot01_bootstrap.py Outdated Show resolved Hide resolved

loader.pyiboot01: use previous var name VIRTENV

9044b31

htgoebel suggested changes Aug 30, 2019

View reviewed changes

PyInstaller/loader/pyiboot01_bootstrap.py Outdated Show resolved Hide resolved

PyInstaller/loader/pyiboot01_bootstrap.py Outdated Show resolved Hide resolved

htgoebel changed the title ~~loader.pyiboot01: add renamed env var to replace VIRTUAL_ENV~~ WIP: add renamed env var to replace VIRTUAL_ENV Aug 30, 2019

htgoebel added the state:needs more work This PR needs more work label Aug 30, 2019

reorx force-pushed the feature/virtual-env-env-var-better-handling branch from a93cd05 to 2de522c Compare August 30, 2019 10:23

reorx changed the title ~~WIP: add renamed env var to replace VIRTUAL_ENV~~ WIP: add options to preserve environment variables Aug 30, 2019

Add news for new options for preserving env vars

04ea52e

htgoebel suggested changes Aug 30, 2019

View reviewed changes

htgoebel reviewed Aug 30, 2019

View reviewed changes

PyInstaller/loader/pyiboot01_bootstrap.py Show resolved Hide resolved

htgoebel reviewed Aug 30, 2019

View reviewed changes

loader.pyiboot01: delete temp vars after using them

842e673

reorx added 4 commits August 31, 2019 16:36

CLI: rename option --pyiboot01-preserved-envs to --disguise-envvar

14f2658

also rename the dest arg name pyiboot01_no_preserved_envs to disguised_envs

analysis: refactor render_pyfile_template to utils

730b1af

new function is called `build_from_template`. other changes: - use re.escape for better readability - add unit test for build_from_template_file

Doc: add description about disguising env vars

bb3177b

Test: --disguise-envvar

0a645b1

reorx requested a review from htgoebel October 31, 2019 12:09

endzyme mentioned this pull request Nov 7, 2019

Sporadic failure initializing helm client with gke clusters FairwindsOps/reckoner#147

Closed

htgoebel removed the state:needs more work This PR needs more work label Nov 15, 2020

Legorooj force-pushed the develop branch 4 times, most recently from 3a8eccd to e20e74c Compare April 17, 2021 04:06

Legorooj force-pushed the develop branch from 1b31d25 to 90431c0 Compare July 13, 2021 10:47

bwoodsend force-pushed the develop branch from edceb66 to 30d949d Compare July 20, 2021 16:01

bwoodsend force-pushed the develop branch from 98913a3 to 51189f6 Compare November 17, 2021 21:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: add options to preserve environment variables #4401

WIP: add options to preserve environment variables #4401

reorx commented Aug 28, 2019 •

edited

reorx commented Aug 30, 2019 •

edited

htgoebel left a comment

htgoebel left a comment

htgoebel Aug 30, 2019

reorx Aug 31, 2019

reorx Aug 31, 2019

reorx commented Aug 30, 2019

htgoebel Aug 30, 2019

htgoebel Aug 30, 2019

reorx Aug 31, 2019

htgoebel commented Aug 30, 2019

reorx commented Aug 31, 2019

reorx commented Oct 31, 2019

WIP: add options to preserve environment variables #4401

Are you sure you want to change the base?

WIP: add options to preserve environment variables #4401

Conversation

reorx commented Aug 28, 2019 • edited

reorx commented Aug 30, 2019 • edited

htgoebel left a comment

Choose a reason for hiding this comment

htgoebel left a comment

Choose a reason for hiding this comment

htgoebel Aug 30, 2019

Choose a reason for hiding this comment

reorx Aug 31, 2019

Choose a reason for hiding this comment

reorx Aug 31, 2019

Choose a reason for hiding this comment

reorx commented Aug 30, 2019

htgoebel Aug 30, 2019

Choose a reason for hiding this comment

htgoebel Aug 30, 2019

Choose a reason for hiding this comment

reorx Aug 31, 2019

Choose a reason for hiding this comment

htgoebel commented Aug 30, 2019

reorx commented Aug 31, 2019

reorx commented Oct 31, 2019

reorx commented Aug 28, 2019 •

edited

reorx commented Aug 30, 2019 •

edited