Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot install eggs and extensions in offline mode #41

Closed
pombredanne opened this issue Jan 9, 2013 · 11 comments

Comments

Projects
None yet
2 participants
@pombredanne
Copy link
Contributor

commented Jan 9, 2013

The code at https://github.com/buildout/buildout/blob/master/src/zc/buildout/buildout.py#L1047 set the dest to None when doing an offline buildout.

As a result, an offline build will fail, trying to join path with an empty dest with AttributeError: 'NoneType' object has no attribute 'endswith' here:
https://github.com/buildout/buildout/blob/master/src/zc/buildout/easy_install.py#L328
newloc = os.path.join(dest, os.path.basename(d.location))

if you set:
dest = buildout_options['eggs-directory']
at https://github.com/buildout/buildout/blob/master/src/zc/buildout/buildout.py#L1047
this solves the problem

The net effect is that its is not possible to run an offline buildout that uses a download-cache where you keep cached downloads. The buildout will attempt to process eggs from the cache alright, but the install step of that eggs will fail, even if all dependencies are satisfied and available in a cache

@pombredanne

This comment has been minimized.

Copy link
Contributor Author

commented Jan 9, 2013

I think this is likely the reason for these issues:
http://google.com/search?q=offline buildout "AttributeError%3A 'NoneType' object has no attribute 'endswith'"
and this bug:
https://bugs.launchpad.net/zc.buildout/+bug/730211

@pombredanne

This comment has been minimized.

Copy link
Contributor Author

commented Jan 9, 2013

There seems of be a similar issue in the zc.egg.recipe
The culprit when using the egg recipe is that the working set invokes install with an explicit None dest:
https://github.com/buildout/buildout/blob/master/src/zc/buildout/easy_install.py#L838

@jimfulton is it possible to boostrap and run a buildout entirely offline at all?

@pombredanne

This comment has been minimized.

Copy link
Contributor Author

commented Jan 9, 2013

well the dest of the install seem to be set to none whenever offline is true though the eggs or source distro are found alright with find-links = file:///some dir with downloads/ and then built in a temp dir... the install only fails becasue the dest=None is passed around

@jimfulton

This comment has been minimized.

Copy link
Member

commented Jan 10, 2013

Unfortunately, "offline" isn't well defined. The original intent was "Don't install new software", which doesn't match the name well. The implementation was biased by this intent. Later the meaning was expanded to "down download things". This matches the name better, but not the implementation and it meant we also couldn't use remote configuration snippets, which broke some use cases of ours.

In any case, we need to decide what offline means and then adjust the implementation if necessary (post 2.0.0 release).

@pombredanne

This comment has been minimized.

Copy link
Contributor Author

commented Jan 10, 2013

@jimfulton thanks for the reply!
By "down download things" did you mean "do not download things"?
My understanding of offline (and I am not the only one there) is "things will work if you have everything available locally and have no network connection".

MO having a working and consistent offline mode with this definition is not too far fetched.
See for instance this quick experiment https://github.com/pombredanne/buildout/commit/ce49c9228eae4091ab5215f76a55d8ad47a013b8 ... this monkey patches the 1.6.x branch to ensure that no call to easy_install.install is made with a dest=None and makes offline working really offline, installing alright if all files are available locally with these settings:

download-cache = downloads
find-links = file:///downloads/

Now, if buildout cannot work consistently offline, what could be the alternative?

pip can work consistently offline with right links but is barely a replacement for only a subset of ez_install and does not handle any configurations. I do not know of an alternative to buildout ;)

So why wait post 2.0.0 release?

@pombredanne pombredanne reopened this Jan 10, 2013

@jimfulton

This comment has been minimized.

Copy link
Member

commented Jan 10, 2013

On Thu, Jan 10, 2013 at 9:21 AM, pombredanne notifications@github.comwrote:

@jimfulton https://github.com/jimfulton thanks for the reply!
By "down download things" did you mean "do not download things"?

Yes. Sorry.

My understanding of offline (and I am not the only one there) is "things
will work if you have everything available locally and have no network
connection".

I get that that is your understanding.

MO having a working and consistent offline mode with this definition is
not too far fetched.
See for instance this quick experiment pombredanne/buildout@ce49c92https://github.com/pombredanne/buildout/commit/ce49c9228eae4091ab5215f76a55d8ad47a013b8... this monkey patches the 1.6.x branch to ensure that no call to
easy_install.install is made with a dest=None and makes offline working
really offline, installing alright if all files are available locally with
these settings:

download-cache = downloads
find-links = file:///downloads/

Now, if buildout cannot work consistently offline, what could be the
alternative?

pip can work consistently offline with right links but is barely a
replacement for only a subset of ez_install and does not handle any
configurations. I do not know of an alternative to buildout ;)

So why wait post 2.0.0 release?

Because:

  1. I'm not working on buildout 1 any more than I have to.

  2. buildout 2 needs to be finished.

  3. Changing the implementation is likely to have far reaching effects.

Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://zo.pe/Kqm

@pombredanne

This comment has been minimized.

Copy link
Contributor Author

commented Jan 10, 2013

@jimfulton

3) Changing the implementation is likely to have far reaching effects.
actually it seems pretty minor from what I can see as only one test is failing (about offline mode) and in zc.recipe.egg
with this (ugly, experimental) monkey patch on the head of master:
https://github.com/pombredanne/buildout/commit/547f9ebbf9d6af6338def7fb0c33071df3bc5682

That said, the alternative for me is to either:

  • try to provide some consistent and comprehensive suite of patches and tests that have a good chance to make it 2.0
  • give up and find something else for the reasons you provided above

Please advise

Cordially
Philippe

@pombredanne

This comment has been minimized.

Copy link
Contributor Author

commented Jan 13, 2013

For reference this is the stack trace when trying to install from cache a cached extension. A similar stack trace comes out for an egg installation too:

  Installing.
  Loading extensions.
  Getting distribution for 'buildout.extensionscripts'.

  Installing.
  Loading extensions.
  Getting distribution for 'buildout.extensionscripts'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/home/pombredanne/tbo/eggs/zc.buildout-1.6.3-py2.6.egg/zc/buildout/buildout.py", line 1852, in main
    getattr(buildout, command)(args)
  File "/home/pombredanne/tbo/eggs/zc.buildout-1.6.3-py2.6.egg/zc/buildout/buildout.py", line 459, in install
    self._load_extensions()
  File "/home/pombredanne/tbo/eggs/zc.buildout-1.6.3-py2.6.egg/zc/buildout/buildout.py", line 1012, in _load_extensions
    prefer_final=not self.accept_buildout_test_releases)
  File "/home/pombredanne/tbo/eggs/zc.buildout-1.6.3-py2.6.egg/zc/buildout/easy_install.py", line 1090, in install
    return installer.install(specs, working_set)
  File "/home/pombredanne/tbo/eggs/zc.buildout-1.6.3-py2.6.egg/zc/buildout/easy_install.py", line 896, in install
    for dist in self._get_dist(requirement, ws, self._always_unzip):
  File "/home/pombredanne/tbo/eggs/zc.buildout-1.6.3-py2.6.egg/zc/buildout/easy_install.py", line 791, in _get_dist
    dist.location, ws, self._dest, dist)
  File "/home/pombredanne/tbo/eggs/zc.buildout-1.6.3-py2.6.egg/zc/buildout/easy_install.py", line 615, in _call_easy_install
    newloc = os.path.join(dest, os.path.basename(d.location))
  File "/home/pombredanne/tbo/lib/python2.6/posixpath.py", line 67, in join
    elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'
@jimfulton

This comment has been minimized.

Copy link
Member

commented Jan 13, 2013

Note that if you want to installe solely from cache, set the install-from-cache option to true:

buildout buildout:install-from-cache=true

@pombredanne

This comment has been minimized.

Copy link
Contributor Author

commented Jan 13, 2013

On Sun, Jan 13, 2013 at 3:17 PM, Jim Fulton notifications@github.comwrote:

Note that if you want to installe solely from cache, set the
install-from-cache option to true:

buildout buildout:install-from-cache=true

Though unless I am getting it all wrong it does not work in offline mode

@jimfulton

This comment has been minimized.

Copy link
Member

commented Jan 14, 2013

Right. Don't use it with offline mode.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.