New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop using xattr to store venvs metadata and centralize it #29

gilgamezh opened this Issue Feb 11, 2015 · 1 comment


None yet
2 participants

gilgamezh commented Feb 11, 2015

some ideas from @rbtcollins

<facubatista> so, fades and xattr :)
<lifeless> righto
<lifeless> so, from my reading of the code
<facubatista> we're using the xattrs to store some metadata: whwere is the venv located, and which dependencies we installed
<lifeless> righto
<facubatista> so next time script is executed we use the same venv, and we quickly decide if there is a new dependency, or any old one needs to be updated/removed
<lifeless> so what about having the venv path be the hash of key inputs (like python version)
<facubatista> so, if not in xattr, where we could store this metadata?
<facubatista> so, like hash([(dependency, revno) for requested_dependencies]) ?
<lifeless> e.g., say you did path = basedir + hash.sha256(''.join([py_ver] + [str(dep) for dep in deps]).hexdigest
<lifeless> yes
<lifeless> would be a different tradeoff in venv setup times
<facubatista> the problem there is that revisions are not that constant, see this example:
<facubatista> you put in a script that you want some module with revno >= 2.0
<facubatista> so, we tell pip to install that dependency, with that revno limitation
<facubatista> pip installs the module, with revno 2.7
<facubatista> we store this 2.7
<facubatista> because that is what is currently there, and in the future we need to react to what was installed
<lifeless> pip freeze will tell you whats installed
<facubatista> so, that changing of revision complicates the hashing
<lifeless> yes
<facubatista> yes, but that involves an external execution... it's slower
<lifeless> I was thinking you'd hash the requirements, not the installed bits
<facubatista> so, question
<lifeless> we can probably get dstufft to export the freeze interface as a library call
<facubatista> if you later edit the script and add a new dependency, it will be a new venv, and we'd need to install all everything again, right?
<facubatista> (don't really think this may happen too frequently)
<lifeless> with my quick sketch, yes
<facubatista> otoh, the benefit of this approach is that if you have two scripts that wants the same dependency, they can reuse the venv
<lifeless> yeah
<lifeless> I don't know if thats really that common or not
<lifeless> but, vim wouldn't break it anymore :)
<facubatista> mmm... probably myself have several scripts that only uses beautifulsoup
<facubatista> or only uses requests
<facubatista> we'd need to put the version too in the hash
<facubatista> so a venv with mod X v1 and other with same mod X but v2 are effectively two different venvs
<lifeless> why?
<lifeless> sorry, I mean
<lifeless> which version
<lifeless> the requirement
<lifeless> or the installed?
<facubatista> the installed one
<facubatista> mmm... we'd need to use the required one
<facubatista> otherwise we won't find the venv later
<lifeless> yeah
<facubatista> any change to requirements will imply a whole venv again
<facubatista> (I don't think that changing requirements will be *that* common)
<lifeless> I can imagine a whole spin-off project
<facubatista> ?
<lifeless> to manage a cache of venvs, introspecting them and renaming rather than rebuilding or things like that
<lifeless> perhapsa double-indirection scheme
<lifeless> $crazy :)(
<facubatista> no that crazy
<facubatista> even, say
<facubatista> with what we talked so far, if we do
<facubatista> first, dependency is mod X (no version), and it installs v5
<facubatista> second, later, you change it to be with version >=3
<facubatista> with the idea discussed, we'll create a new venv
<facubatista> but, what if...
<facubatista> we have somewhere (a fades DB) where we could ask for those requirements, and it will say "I have this venv, use it"
<facubatista> other script could say, for "mod X == 5", and it will still use the same one
<facubatista> so, no more hashes to the name... the first time we just create a new venv, install what is requested, and then store somewhere "this venv has this stuff installed"
<lifeless> sounds nice
<facubatista> next time, we go to the DB and check for a venv that satisfies the requirements... if yes, use it; if not, create a new one
<facubatista> no matter if it was created for the same script, or not
<facubatista> mmm... I like this, eh!  I'm not even struggling with the feeling of "oh, we need to dump all that code" :p
<lifeless> cool, write one to throw away!
<facubatista> indeed
<facubatista> where you wanted to use fades that xattr was a problem?
<lifeless> couple places; firstly I've a windows machine now (mainly for games, but I Python on it when I'm checking upstream patches to make sure I don't break the windows buildbots)
<lifeless> secondly, I use vim :)
<lifeless> I've also had some weird experiences with xattr here and there
<lifeless> I don't really trust them, because they get lost so easily (e.g. by vim) - the standardish atomic idioms don't work
<lifeless> its not that I think the kernel is buggy or anything
<lifeless> just that their semantics don't match the semantics most unix programs use around file management
<facubatista> yes, vim has a problem with xattr (*I* opened them a bug about this :p ), there's a hack, but better if we don't use it
<facubatista> the xattrs semantics *suck*
<facubatista> they're even complicated to use in Python! and that's the main reason I didn't offer a patch for vim :p
<facubatista> and thanks for all the ideas!!

@gilgamezh gilgamezh added this to the Second release milestone Feb 13, 2015


This comment has been minimized.

Show comment
Hide comment

facundobatista Feb 14, 2015


Yes! It should :p; adjusting issue name to reflect this


facundobatista commented Feb 14, 2015

Yes! It should :p; adjusting issue name to reflect this

@facundobatista facundobatista changed the title from should fades quit xattr? to Stop using xattr to store venvs metadata and centralize it Feb 14, 2015

@facundobatista facundobatista self-assigned this Feb 15, 2015

@facundobatista facundobatista referenced this issue Feb 22, 2015


Cache #35

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment