Stop using xattr to store venvs metadata and centralize it

[gilgamezh]

some ideas from @rbtcollins

http://linkode.org/ujwxKV00Qyal1haoQRzkg5

<facubatista> so, fades and xattr :)
<lifeless> righto
<lifeless> so, from my reading of the code
<facubatista> we're using the xattrs to store some metadata: whwere is the venv located, and which dependencies we installed
<lifeless> righto
<facubatista> so next time script is executed we use the same venv, and we quickly decide if there is a new dependency, or any old one needs to be updated/removed
<lifeless> so what about having the venv path be the hash of key inputs (like python version)
<facubatista> so, if not in xattr, where we could store this metadata?
<facubatista> so, like hash([(dependency, revno) for requested_dependencies]) ?
<lifeless> e.g., say you did path = basedir + hash.sha256(''.join([py_ver] + [str(dep) for dep in deps]).hexdigest
<lifeless> yes
<lifeless> would be a different tradeoff in venv setup times
<facubatista> the problem there is that revisions are not that constant, see this example:
<facubatista> you put in a script that you want some module with revno >= 2.0
<facubatista> so, we tell pip to install that dependency, with that revno limitation
<facubatista> pip installs the module, with revno 2.7
<facubatista> we store this 2.7
<facubatista> because that is what is currently there, and in the future we need to react to what was installed
<lifeless> pip freeze will tell you whats installed
<facubatista> so, that changing of revision complicates the hashing
<lifeless> yes
<facubatista> yes, but that involves an external execution... it's slower
<lifeless> I was thinking you'd hash the requirements, not the installed bits
<facubatista> so, question
<lifeless> we can probably get dstufft to export the freeze interface as a library call
<facubatista> if you later edit the script and add a new dependency, it will be a new venv, and we'd need to install all everything again, right?
<facubatista> (don't really think this may happen too frequently)
<lifeless> with my quick sketch, yes
<facubatista> otoh, the benefit of this approach is that if you have two scripts that wants the same dependency, they can reuse the venv
<lifeless> yeah
<lifeless> I don't know if thats really that common or not
<lifeless> but, vim wouldn't break it anymore :)
<facubatista> mmm... probably myself have several scripts that only uses beautifulsoup
<facubatista> or only uses requests
<facubatista> we'd need to put the version too in the hash
<facubatista> so a venv with mod X v1 and other with same mod X but v2 are effectively two different venvs
<lifeless> why?
<lifeless> sorry, I mean
<lifeless> which version
<lifeless> the requirement
<lifeless> or the installed?
<facubatista> the installed one
<facubatista> mmm... we'd need to use the required one
<facubatista> otherwise we won't find the venv later
<lifeless> yeah
<facubatista> any change to requirements will imply a whole venv again
<facubatista> (I don't think that changing requirements will be *that* common)
<lifeless> I can imagine a whole spin-off project
<facubatista> ?
<lifeless> to manage a cache of venvs, introspecting them and renaming rather than rebuilding or things like that
<lifeless> perhapsa double-indirection scheme
<lifeless> $crazy :)(
<facubatista> no that crazy
<facubatista> even, say
<facubatista> with what we talked so far, if we do
<facubatista> first, dependency is mod X (no version), and it installs v5
<facubatista> second, later, you change it to be with version >=3
<facubatista> with the idea discussed, we'll create a new venv
<facubatista> but, what if...
<facubatista> we have somewhere (a fades DB) where we could ask for those requirements, and it will say "I have this venv, use it"
<facubatista> other script could say, for "mod X == 5", and it will still use the same one
<facubatista> so, no more hashes to the name... the first time we just create a new venv, install what is requested, and then store somewhere "this venv has this stuff installed"
<lifeless> sounds nice
<facubatista> next time, we go to the DB and check for a venv that satisfies the requirements... if yes, use it; if not, create a new one
<facubatista> no matter if it was created for the same script, or not
<facubatista> mmm... I like this, eh!  I'm not even struggling with the feeling of "oh, we need to dump all that code" :p
<lifeless> cool, write one to throw away!
<facubatista> indeed
<facubatista> where you wanted to use fades that xattr was a problem?
<lifeless> couple places; firstly I've a windows machine now (mainly for games, but I Python on it when I'm checking upstream patches to make sure I don't break the windows buildbots)
<lifeless> secondly, I use vim :)
<lifeless> I've also had some weird experiences with xattr here and there
<lifeless> I don't really trust them, because they get lost so easily (e.g. by vim) - the standardish atomic idioms don't work
<lifeless> its not that I think the kernel is buggy or anything
<lifeless> just that their semantics don't match the semantics most unix programs use around file management
<facubatista> yes, vim has a problem with xattr (*I* opened them a bug about this :p ), there's a hack, but better if we don't use it
<facubatista> the xattrs semantics *suck*
<facubatista> they're even complicated to use in Python! and that's the main reason I didn't offer a patch for vim :p
<facubatista> and thanks for all the ideas!!

[facundobatista]

Yes! It should :p; adjusting issue name to reflect this