Cleanup other directories in .shiv #88
Comments
|
hey @rsyring! The lack of robustness in how shiv handles it's cache directory is something I think about regularly. I think there's a lot of room for improvement but it's a dangerous area because I think that folks (myself included) make a lot of assumptions about environments. I like your idea a lot, how would you feel about this being implemented as an environment variable as opposed to a build-time argument? The reason I prefer environment variable is that it makes it opt-in on the runtime rather than locked in at build time. Let me know what you think! |
|
@lorencarvalho thanks for the quick reply. I would prefer that to nothing. However, I could see that be pretty annoying to communicate to users who might be using a binary, especially if they are non-technical. That is, in some scenarios, mine included, I actually want to lock it in at build time. Maybe allow both: I think the first two options are obvious. The last one would be what you proposed and dependent at run time on some pre-determined environment var: Feedback welcome. Thanks again. |
|
hey @rsyring, Sorry for the delay in getting back to you -- been thinking on how best to solve this. How would you feel about a generic Currently the point at which this preamble is executed doesn't have access to the packed environment, but that's just a matter of improving the implementation. Let me know what you think! |
|
@lorencarvalho Thanks for the write up. I'd be happy to use the preamble to do the environment cleanup. Although, I still think there is a good argument for having this particular feature built-in since it's how shiv works that creates the "mess" shiv should make it easy to also clean things up. Although, I'm sympathetic to the concern you have about making environment assumptions, I'd like to point out:
Thanks for your reply, not trying to be difficult. If you decide to go the preamble route, maybe we could give an example preamble which does cleanup. |
|
Is there any progress with this issue? Saw that shiv was changing cache signatures, which makes it harder to follow up... I am using following for cleanup: (cd ~/.shiv && for pyz in $(ls -1d * | grep "_" | sed 's/_[^\_]*//' | uniq); do for oldpyz in $(ls -1drt ${pyz}_* | head -n -1); do rm -rf $oldpyz; done; done) |
|
hey @diefans, yes we did change the dir names recently (from being a uuid to a sha256 of the pyz's contents), sorry about that! I don't think we'll be implementing cleanup of the ~/.shiv directory in shiv itself, but I would certainly be open to an additional entry point (similar to the |
|
FWIW, I still feel like if shiv is creating the problem, shiv is responsible for providing a good solution for doing the cleanup. And when I say shiv, I really mean the binaries that shiv is building. Keep in mind that we are creating the binaries often b/c we want to ship them to systems where we don't have full python dev environments. So some kind of Was there any particular objection to something like:
Default to "never" and mark this feature as experimental and see how it pans out? If it works well, maybe you set the default to "always" eventually. FWIW, I created a script to do the cleanup: import json
import logging
import pathlib
import shutil
import os
from shiv.bootstrap import cache_path, current_zipfile
from shiv.bootstrap.environment import Environment
log = logging.getLogger(__name__)
def simulate_bootstrap():
# get a handle of the currently executing zip file
archive = current_zipfile()
# create an environment object (a combination of env vars and json metadata)
env = Environment.from_json(archive.read("environment.json").decode())
# Store information into the environment for "shiv aware" applications
os.environ['_SHIV_INFO_JSON'] = '''{{
"cache_path": "{cache_dpath}",
"environ": {env_json_str}
}}'''.strip().format(
cache_dpath=cache_path(archive, env.root, env.build_id),
env_json_str=env.to_json()
)
def cleanup_shivs():
# Simulate what shiv.bootstrap:bootstrap() could do to help put pertient information in the
# environment to help "shiv aware" applications save time by not having to reprocess what shiv
# has already done. Not needed if shiv does the cleanup.
simulate_bootstrap()
shiv_info_json = os.environ.get('_SHIV_INFO_JSON')
if not shiv_info_json:
log.debug('Not running from shiv bootstrap')
return
shiv_info = json.loads(shiv_info_json)
env = shiv_info['environ']
cache_dpath = pathlib.Path(shiv_info['cache_path'])
build_id = env['build_id']
dname_prefix = cache_dpath.name[0:-64]
dname_length = len(cache_dpath.name)
cache_root_dpath = cache_dpath.parent
for dpath in cache_root_dpath.iterdir():
dir_name = dpath.name
if build_id in dir_name \
or len(dir_name) != dname_length \
or dir_name[0:-64] != dname_prefix \
or not dpath.is_dir():
continue
log.debug(f'Deleting {dpath} and lock file')
shutil.rmtree(dpath)
lock_fname = f'.{dpath.stem}_lock'
pathlib.Path(cache_root_dpath, lock_fname).unlink(missing_ok=True)If you are interested, I can do some legwork to get this tested, documented, and put up a PR. Thanks. |
|
Hey @rsyring, I included an example script that cleans up old execution dirs: Theoretically you could even pack another pyz into the pyz (though I haven't tested this). Let me know what you think, if it works for you I will add some tests and cut a PR. |
We're currently testing out
It hadn't occurred to me that a preamble could solve both problems 💡 We'd love to see this feature added, thanks! |
|
@luislew that's awesome to hear! say hi to @borgstrom for me :) I went ahead and cut a PR with this feature, take it for a spin and let me know what you think :) |
|
Closing this issue, the |
I'd like to be able to have a shiv option that would instruct the bootstrap code to clean out old directories in .shiv for the current app. Maybe:
With that option, when the first bootstrap of a build finished, the boostrap code would essentially run:
Excluding the folder that contains the files for the currently running app.
The text was updated successfully, but these errors were encountered: