Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Empty dependency __pycache__ folders left behind on upgrade #6743

Open
legoktm opened this issue Feb 6, 2023 · 3 comments · May be fixed by #6782
Open

Empty dependency __pycache__ folders left behind on upgrade #6743

legoktm opened this issue Feb 6, 2023 · 3 comments · May be fixed by #6782
Assignees
@legoktm

Comments

@legoktm
Copy link
Member

legoktm commented Feb 6, 2023 •
edited

Description

When upgrading from 2.5.1 to 2.6.0~rc1 (current develop), dpkg emits warnings that it can't remove the passlib directories because of the __pycache__ directories.

Steps to Reproduce

  • Install 2.5.1 on prod/staging
  • make build-debs on develop
  • Install the new securedrop-app-code deb

Expected Behavior

No dpkg warnings; /opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ is deleted

Actual Behavior

Unpacking securedrop-app-code (2.6.0~rc1+focal) over (2.5.1+focal) ...                                                 
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils/compat/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils/compat': Directory not empty                                                                                          
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/tests/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/tests': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/handlers/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/handlers': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext/django/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext/django': Directory not empty                                                                                            
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/scrypt/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/scrypt': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/_blowfish/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/_blowfish': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/__pycache__': Directory not empty                                                                                    
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/_setup/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/_setup': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/__pycache__': Directory not empty                                                                                           
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib': Directory not empty

Comments

All of the code is correctly removed, it's just the empty folders left behind...

sdadmin@app:~$ find /opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils/compat
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils/compat/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/tests
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/tests/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/scrypt
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/scrypt/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/_blowfish
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/_blowfish/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext/django
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext/django/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/_setup
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/_setup/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/handlers
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/handlers/__pycache__
@legoktm legoktm added this to the 2.6.0 milestone Feb 6, 2023
@legoktm legoktm changed the title Dependency __pycache__ left behind on upgrades Empty dependency __pycache__ folders left behind on upgrade Feb 6, 2023
@legoktm legoktm mentioned this issue Feb 6, 2023
Merged
5 tasks
@legoktm
Copy link
Member Author

legoktm commented Feb 6, 2023

To clarify, there is no actual problem caused by this, it's just messy and an unintentional consequence of having the postinst generate the __pycache__ folders.

@legoktm
Copy link
Member Author

legoktm commented Feb 7, 2023 •
edited

Looking at what dh-python does, we need a prerm script that runs py3clean; see https://salsa.debian.org/python-team/tools/dh-python/-/blob/master/autoscripts/prerm-py3clean

So the flow ends up being:

  • first install triggers postinst - runs py3compile to generate cache
  • upgrade triggers:
    • prerm (before new version is unpacked) - runs py3clean to delete existing cache
    • postinst (after new version is in place) - runs py3 compile to generate cache from scratch

@legoktm legoktm self-assigned this Feb 7, 2023
legoktm added a commit that referenced this issue Apr 14, 2023
@legoktm
Clean Python cache in prerm only
fbcd77b 
We are currently cleaning the Python cache in the postinst and postrm,
but that still leaves us with warnings from apt that non-empty
`__pycache__` directories couldn't be deleted.

This mimics what dh-python does, which is to run py3compile in the
postinst (already doing) and run py3clean in the prerm.

During a fresh install, only the postinst is triggered to compile the
cache.

During an upgrade, first the prerm runs, removing the cache, then the
new version is unpacked, then postinst goes, recreating the cache.

Fixes #6743.
@legoktm legoktm linked a pull request Apr 14, 2023 that will close this issue
Draft
9 tasks
@legoktm
Copy link
Member Author

legoktm commented May 15, 2023

[Backlog pruning, 5/15] Removing this from 2.6.0, it's not important enough.

@legoktm legoktm removed this from the SecureDrop 2.6.0 milestone May 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@legoktm legoktm

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Clean Python cache in prerm only freedomofpress/securedrop
1 participant
@legoktm