New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The remote end hung up unexpectedly #207

Closed
saxejeffreyd opened this Issue Dec 5, 2013 · 45 comments

Comments

Projects
None yet
5 participants
@saxejeffreyd

saxejeffreyd commented Dec 5, 2013

I'm attempting to install bcbio-nextgen. It goes through quite a bit, but then I see:

Requirement already satisfied (use --upgrade to upgrade): ecdsa in /opt/az/local/bcbio-nextgen/use_wget/installdir/anaconda/lib/python2.7/site-packages (from paramiko>=1.10.0->fabric>=1.7.0->-r https://raw.github.com/chapmanb/bcbio-nextgen/master/requirements.txt (line 6))
Cleaning up...
Installing data and third party dependencies
Upgrading third party tools to latest versions
Cloning into 'cloudbiolinux'...
remote: Counting objects: 8294, done.
error: RPC failed; result=18, HTTP code = 200
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
Traceback (most recent call last):
File "/opt/az/local/bcbio-nextgen/use_wget/installdir/anaconda/bin/bcbio_nextgen.py", line 43, in
install.upgrade_bcbio(kwargs["args"])
File "/opt/az/local/bcbio-nextgen/use_wget/installdir/anaconda/lib/python2.7/site-packages/bcbio/install.py", line 49, in upgrade_bcbio
upgrade_thirdparty_tools(args, REMOTES)
File "/opt/az/local/bcbio-nextgen/use_wget/installdir/anaconda/lib/python2.7/site-packages/bcbio/install.py", line 157, in upgrade_thirdparty_tools
cbl = get_cloudbiolinux(remotes)
File "/opt/az/local/bcbio-nextgen/use_wget/installdir/anaconda/lib/python2.7/site-packages/bcbio/install.py", line 270, in get_cloudbiolinux
subprocess.check_call(["git", "clone", remotes["cloudbiolinux"]])
File "/opt/az/local/bcbio-nextgen/use_wget/installdir/anaconda/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'clone', 'https://github.com/chapmanb/cloudbiolinux.git']' returned non-zero exit status 128
Traceback (most recent call last):
File "bcbio_nextgen_install.py", line 226, in
main(parser.parse_args(), sys.argv[1:])
File "bcbio_nextgen_install.py", line 36, in main
subprocess.check_call([bcbio["bcbio_nextgen.py"], "upgrade"] + _clean_args(sys_argv, args, bcbio))
File "/opt/az/local/bcbio-nextgen/jakub/bcbio-nextgen/anaconda/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/az/local/bcbio-nextgen/use_wget/installdir/anaconda/bin/bcbio_nextgen.py', 'upgrade', '--tooldir=/opt/az/local/bcbio-nextgen/use_wget/tooldir', '--data']' returned non-zero exit status 1

This has been happening the past 2 days. I originally thought that it was due to network problems here, but they claim to have fixed our network issues and this is still occurring.

Jeff

@roryk

This comment has been minimized.

Collaborator

roryk commented Dec 5, 2013

Hi Jeff,

This thread on SO seems like it might have some ideas, especially the comment on the bottom:

http://stackoverflow.com/questions/12236415/git-clone-return-result-18-code-200-on-a-specific-repository

Does any of that help?

@chapmanb

This comment has been minimized.

Member

chapmanb commented Dec 5, 2013

Jeff;
Thanks for trying bcbio-nextgen and sorry about the install problems. It sounds like there is something problematic with cloning from GitHub. Does:

git clone https://github.com/chapmanb/cloudbiolinux.git

work from the command line? Searching stack overflow, it seems like this could be a system specific CURL problem (http://stackoverflow.com/a/17981885/252589). Could you try:

git config --global http.postBuffer 524288000

If that works, we can add this to the documentation. Thanks much.

@chapmanb

This comment has been minimized.

Member

chapmanb commented Dec 5, 2013

Rory -- nice same time, same idea answer.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 5, 2013

Hi Brad and Rory,

I tried

git config --global http.postBuffer 524288000

but unfortunately it made no difference. I'm now trying

git clone https://github.com/chapmanb/cloudbiolinux.git

I'll let you know what happens.

Jeff

On Thu, Dec 5, 2013 at 9:55 AM, Brad Chapman notifications@github.comwrote:

Jeff;
Thanks for trying bcbio-nextgen and sorry about the install problems. It
sounds like there is something problematic with cloning from GitHub. Does:

git clone https://github.com/chapmanb/cloudbiolinux.git

work from the command line? Searching stack overflow, it seems like this
could be a system specific CURL problem (
http://stackoverflow.com/a/17981885/252589). Could you try:

git config --global http.postBuffer 524288000

If that works, we can add this to the documentation. Thanks much.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-29903692
.

@roryk

This comment has been minimized.

Collaborator

roryk commented Dec 6, 2013

Hi Jeff did cloning the repository directory work?

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 6, 2013

Not yet. Still having problems.

Sara said she thought you had already pulled everything down to your own
account on our system. Is that true? If so, I have some questions.

  1. Why am I doing it if it's already here?
  2. How did you do it, e.g. what command did you use, what proxy did you
    use, etc? When I get into work I can look at all the settings I have in my
    environment and send them.

Jeff

On Thu, Dec 5, 2013 at 10:45 PM, Rory Kirchner notifications@github.comwrote:

Hi Jeff did cloning the repository directory work?


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-29961620
.

@mjafin

This comment has been minimized.

Contributor

mjafin commented Dec 6, 2013

Hi Jeff,

  1. I think Rory started the install but has not had the chance to check up on it. I don't think it finished, looking at the folder on our system. We need to be able to do the install ourselves in an automatic manner because we need to pull the latest version very frequently.
  2. I managed to pull CloudBioLinux on rask today successfully, so I will send you my proxy settings etc.
@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 6, 2013

Sorry I didn't reply earlier. It appears that our wonderful network is the
problem. We'll see if I have any luck next week.

You have no idea how much I despise our network.

Jeff

On Thu, Dec 5, 2013 at 10:45 PM, Rory Kirchner notifications@github.comwrote:

Hi Jeff did cloning the repository directory work?


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-29961620
.

@roryk

This comment has been minimized.

Collaborator

roryk commented Dec 7, 2013

hehe-- bummer about the network but good that it is a resolvable issue. Thanks!

@roryk roryk closed this Dec 7, 2013

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 10, 2013

Just to let you know, yes, all of the problems I had last week were due to
problems with our network. When I tried again Monday morning everything
immediately started downloading, and I now have everything.

However, I now have a question about installing something else which I
believe is also yours. Justin Johnson has asked me to install gemini. I
followed directions that I found at
http://gemini.readthedocs.org/en/latest/content/installation.html
I did
$ wget
https://raw.github.com/arq5x/gemini/master/gemini/scripts/gemini_install.py
But now I get:

[root@green gemini]# python gemini_install.py
/opt/az/local/gemini/installdir /opt/az/local/gemini/installdir/share/gemini
Checking required dependencies...
git found
wget found
Traceback (most recent call last):
File "gemini_install.py", line 248, in
main(parser.parse_args())
File "gemini_install.py", line 34, in main
check_dependencies()
File "gemini_install.py", line 228, in check_dependencies
subprocess.check_call([cmd, "--version"], stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
File "/opt/az/local/psf/python/64_2.7.3/lib/python2.7/subprocess.py",
line 511, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['curl', '--version']' returned
non-zero exit status 2
[root@green gemini]#

Do you have any idea what may be causing this?

Jeff

On Thu, Dec 5, 2013 at 9:56 AM, Brad Chapman notifications@github.comwrote:

Rory -- nice same time, same idea answer.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-29903745
.

@roryk

This comment has been minimized.

Collaborator

roryk commented Dec 10, 2013

Hi Jeff,

That is excellent news! GEMINI isn't installed by default because it has a huge amount of files to download. You can install it like this:

bcbio_nextgen.py upgrade -u development --toolsplus gemini

@chapmanb

This comment has been minimized.

Member

chapmanb commented Dec 10, 2013

Jeff;
Seconded on the awesome success. Thanks for working through this. The command changed a little in the latest version:

bcbio_nextgen.py upgrade --tools --toolsplus gemini

so you don't need to manually install. I'm not sure if you'll run into the same error, but your message looks like something is problematic with curl. What does curl --version give on the machine?

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 10, 2013

I tried the commands that you both sent me (see below) but both complain
about '--toolsplus gemini'

Jeff

[root@green ~]# bcbio_nextgen.py upgrade --tools --toolsplus gemini
usage: bcbio_nextgen.py [-h] {upgrade} ...
bcbio_nextgen.py: error: unrecognized arguments: --toolsplus gemini
[root@green ~]# bcbio_nextgen.py upgrade -u development --toolsplus gemini
usage: bcbio_nextgen.py [-h] {upgrade} ...
bcbio_nextgen.py: error: unrecognized arguments: --toolsplus gemini

On Tue, Dec 10, 2013 at 1:57 PM, Brad Chapman notifications@github.comwrote:

Jeff;
Seconded on the awesome success. Thanks for the working through this. The
command changed a little in the latest version:

bcbio_nextgen.py upgrade --tools --toolsplus gemini

so you don't need to manually install. I'm not sure if you'll run into the
same error, but your message looks like something is problematic with curl.
What does curl --version give on the machine?


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-30256539
.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 10, 2013

Brad,

[root@green bin]# curl --version
curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b
zlib/1.2.7 libidn/0.6.5
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
[root@green bin]#

On Tue, Dec 10, 2013 at 1:57 PM, Brad Chapman notifications@github.comwrote:

Jeff;
Seconded on the awesome success. Thanks for the working through this. The
command changed a little in the latest version:

bcbio_nextgen.py upgrade --tools --toolsplus gemini

so you don't need to manually install. I'm not sure if you'll run into the
same error, but your message looks like something is problematic with curl.
What does curl --version give on the machine?


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-30256539
.

@chapmanb

This comment has been minimized.

Member

chapmanb commented Dec 11, 2013

Jeff;
Sorry we had a typo above, it should be --toolplus (no s), so:

 bcbio_nextgen.py upgrade --tools --toolplus gemini

Hopefully that'll work cleanly. I'm totally stumped why gemini errored out with curl but will keep my fingers crossed it was a one off error.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 11, 2013

Brad,

Now I get:

[root@green use_git_clone]# bcbio_nextgen.py upgrade --tools --toolplus
gemini
usage: bcbio_nextgen.py upgrade [-h] [--tooldir TOOLDIR] [--tools]
[-u {stable,development,system,skip}]
[--toolplus {protected,data}]
[--genomes GENOMES] [--aligners ALIGNERS]
[--data] [--nosudo] [--isolate]
[--tooldist {minimal,full}]
[--distribution
{ubuntu,debian,centos,scientificlinux,macosx}]
bcbio_nextgen.py upgrade: error: argument --toolplus: invalid choice:
'gemini' (choose from 'protected', 'data')
[root@green use_git_clone]#

Trying the other method still gives problems with curl. That may well be
due to our proxy/firewall. I have http_proxy and ftp_proxy defined. Still
I get:

[root@green gemini]# python gemini_install.py
/opt/az/local/gemini/installdir
/opt/az/local/gemini/installdir/share/gemini
Checking required dependencies...
git found
wget found
Traceback (most recent call last):
File "gemini_install.py", line 248, in
main(parser.parse_args())
File "gemini_install.py", line 34, in main
check_dependencies()
File "gemini_install.py", line 228, in check_dependencies
subprocess.check_call([cmd, "--version"], stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
File
"/opt/az/local/bcbio-nextgen/use_git_clone/installdir/anaconda/lib/python2.7/subprocess.py",
line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['curl', '--version']' returned
non-zero exit status 2
[root@green gemini]# curl --version
curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b
zlib/1.2.7 libidn/0.6.5
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
[root@green gemini]#

Thanks.

Jeff

On Tue, Dec 10, 2013 at 8:00 PM, Brad Chapman notifications@github.comwrote:

Jeff;
Sorry we had a typo above, it should be --toolplus (no s), so:

bcbio_nextgen.py upgrade --tools --toolplus gemini

Hopefully that'll work cleanly. I'm totally stumped why gemini errored out
with curl but will keep my fingers crossed it was a one off error.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-30284876
.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 11, 2013

Brad,

I'm looking at
https://bcbio-nextgen.readthedocs.org/en/latest/contents/installation.html#automated.
There they show

bcbio_nextgen.py upgrade -u stable --tools --data

Under --toolplus it shows:

data Data rich supplemental tools. A good example is
GEMINI, which provides rich annotation of variant calls
but requires download of external data sources.

So I attempted

bcbio_nextgen.py upgrade -u development --toolplus data

This did do something, telling me it was downloading conda-2.2.4 and
conda-2.2.3. How can I tell if gemini is installed somewhere? I
didn't see any mention of it.

Jeff

On Tue, Dec 10, 2013 at 8:00 PM, Brad Chapman notifications@github.comwrote:

Jeff;
Sorry we had a typo above, it should be --toolplus (no s), so:

bcbio_nextgen.py upgrade --tools --toolplus gemini

Hopefully that'll work cleanly. I'm totally stumped why gemini errored out
with curl but will keep my fingers crossed it was a one off error.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-30284876
.

chapmanb added a commit to arq5x/gemini that referenced this issue Dec 11, 2013

@chapmanb

This comment has been minimized.

Member

chapmanb commented Dec 11, 2013

Jeff;
Sigh, sorry about all the typos. I suck. I was travelling yesterday and guess I shouldn't try to help from the train. Apologies for so many wild goose chases. You're right, it should be:

bcbio_nextgen.py upgrade --tools --toolplus data

This is telling it: update the tools, and also add in the extra data category (which is just gemini right now). In your command without the --tools flag it won't try to upgrade tools so that's the only change you need from what you did. I also pushed a fix to the gemini installer to avoid the curl errors you were reporting. It won't fail now on non-zero exit codes, so hopefully will go cleanly.

Thanks again for all the patience and hope this goes for you now without any issues.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 11, 2013

Hi Brad,

Okay, I got the new gemini_install.py and that gets farther. It appears
that this downloads much of the same stuff that I got with
bcbio_nextgen_install.py. Apparently I should be directing this all to the
same installation directory as was used by bcbio-nextgen. Is that correct?

Also, as I mentioned, it gets farther. It didn't complete. I get

Installing collected packages: setuptools, distribute
Found existing installation: setuptools 1.4
Cleaning up...
Cannot remove entries from nonexistent file
/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/easy-install.pth
Storing complete log in /root/.pip/pip.log
Traceback (most recent call last):
File "gemini_install.py", line 251, in
main(parser.parse_args())
File "gemini_install.py", line 44, in main
gemini = install_gemini(anaconda, remotes, args.datadir, args.tooldir,
args.sudo)
File "gemini_install.py", line 73, in install_gemini
subprocess.check_call([anaconda["pip"], "install", "--upgrade",
"distribute"])
File
"/opt/az/local/bcbio-nextgen/use_git_clone/installdir/anaconda/lib/python2.7/subprocess.py",
line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command
'['/opt/az/local/gemini/installdir/share/gemini/anaconda/bin/pip',
'install', '--upgrade', 'distribute']' returned non-zero exit status 1
[root@green gemini]#

Maybe it's essentially done, just died trying to clean things up at the
end. I'm not sure.

Jeff

On Wed, Dec 11, 2013 at 9:41 AM, Brad Chapman notifications@github.comwrote:

Jeff;
Sigh, sorry about all the typos. I suck. I was travelling yesterday and
guess I shouldn't try to help from the train. Apologies for so many wild
goose chases. You're right, it should be:

bcbio_nextgen.py upgrade --tools --toolplus data

This is telling it: update the tools, and also add in the extra datacategory (which is just gemini right now). In your command without the
--tools flag it won't try to upgrade tools so that's the only change you
need from what you did. I also pushed a fix to the gemini installer to
avoid the curl errors you were reporting. It won't fail now on non-zero
exit codes, so hopefully will go cleanly.

Thanks again for all the patience and hope this goes for you now without
any issues.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-30325052
.

chapmanb added a commit to arq5x/gemini that referenced this issue Dec 11, 2013

@chapmanb

This comment has been minimized.

Member

chapmanb commented Dec 11, 2013

Jeff;
Sorry, it looks like another pip problem I wasn't successfully working around. I pushed a new fix here:

wget https://raw.github.com/arq5x/gemini/master/gemini/scripts/gemini_install.py

which will hopefully keep it going cleanly if you redownload and restart. Sorry about the problems.

In terms of what it installs, it will do a lot of the same stuff that bcbio-nextgen does but is an isolated program. I wrote the installers for both so they borrow a lot but it's easier to keep it as a separate tool rather than having a bcbio-nextgen only process. It still does have a way to go since it'll install remaining data which are some fairly hefty downloads. Hopefully it will keep chugging with this latest fix. Let me know if you run into anything else at all and thanks again for pushing this through.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 11, 2013

Hi Brad,

Getting closer. I'm getting timed out. I don't know if that's our
wonderful network again, or due to something else.

Jeff

Requirement already satisfied (use --upgrade to upgrade): PyYAML>=3.10 in
/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages
(from -r https://raw.github.com/arq5x/gemini/master/requirements.txt (line
5))
Downloading/unpacking pybedtools>=0.6.2 (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 6))
Error The read operation timed out while getting
https://pypi.python.org/packages/source/p/pybedtools/pybedtools-0.6.2.tar.gz#md5=ee3677da171973bf4299242efad62685(from
https://pypi.python.org/simple/pybedtools/)
Cleaning up...
Exception:
Traceback (most recent call last):
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/pip/basecommand.py",
line 134, in main
status = self.run(options, args)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/pip/commands/install.py",
line 236, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle,
bundle=self.bundle)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/pip/req.py",
line 1092, in prepare_files
self.unpack_url(url, location, self.is_download)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/pip/req.py",
line 1238, in unpack_url
retval = unpack_http_url(link, location, self.download_cache,
self.download_dir)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/pip/download.py",
line 602, in unpack_http_url
resp = _get_response_from_url(target_url, link)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/pip/download.py",
line 638, in _get_response_from_url
resp = urlopen(target_url)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/pip/download.py",
line 176, in call
response = self.get_opener(scheme=scheme).open(url)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/urllib2.py",
line 404, in open
response = self._open(req, data)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/urllib2.py",
line 422, in _open
'_open', req)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/urllib2.py",
line 382, in _call_chain
result = func(*args)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/pip/download.py",
line 155, in https_open
return self.do_open(self.specialized_conn_class, req)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/urllib2.py",
line 1187, in do_open
r = h.getresponse(buffering=True)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/httplib.py",
line 1045, in getresponse
response.begin()
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/httplib.py",
line 409, in begin
version, status, reason = self._read_status()
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/httplib.py",
line 365, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/socket.py",
line 476, in readline
data = self._sock.recv(self._rbufsize)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/ssl.py",
line 241, in recv
return self.read(buflen)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/ssl.py",
line 160, in read
return self._sslobj.read(len)
SSLError: The read operation timed out

Storing complete log in /root/.pip/pip.log
Traceback (most recent call last):
File "gemini_install.py", line 254, in
main(parser.parse_args())
File "gemini_install.py", line 44, in main
gemini = install_gemini(anaconda, remotes, args.datadir, args.tooldir,
args.sudo)
File "gemini_install.py", line 82, in install_gemini
subprocess.check_call([anaconda["pip"], "install", "-r",
remotes["requirements"]])
File
"/opt/az/local/bcbio-nextgen/use_git_clone/installdir/anaconda/lib/python2.7/subprocess.py",
line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command
'['/opt/az/local/gemini/installdir/share/gemini/anaconda/bin/pip',
'install', '-r', '
https://raw.github.com/arq5x/gemini/master/requirements.txt']' returned
non-zero exit status 2
[root@green gemini]#

On Wed, Dec 11, 2013 at 10:56 AM, Brad Chapman notifications@github.comwrote:

Jeff;
Sorry, it looks like another pip problem I wasn't successfully working
around. I pushed a new fix here:

wget https://raw.github.com/arq5x/gemini/master/gemini/scripts/gemini_install.py

which will hopefully keep it going cleanly if you redownload and restart.
Sorry about the problems.

In terms of what it installs, it will do a lot of the same stuff that
bcbio-nextgen does but is an isolated program. I wrote the installers for
both so they borrow a lot but it's easier to keep it as a separate tool
rather than having a bcbio-nextgen only process. It still does have a way
to go since it'll install remaining data which are some fairly hefty
downloads. Hopefully it will keep chugging with this latest fix. Let me
know if you run into anything else at all and thanks again for pushing this
through.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-30332212
.

@chapmanb

This comment has been minimized.

Member

chapmanb commented Dec 11, 2013

Jeff;
Unfortunately I don't have a better workaround for this one than trying again to see if it's persistent. It looks like a one off network error (which could also be due to pypi). pybedtools is also used inside bcbio-nextgen so this "should work" on the network since it had to be done once during the original bcbio-nextgen install. I hope retry magic will work. Crossing fingers.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 11, 2013

Hi Brad,

I do believe our network is causing problems again. I've tried different
proxies. I'm now back to the 'curl' error: Could this be the same problem
as before, but in a different place?

Jeff

INFO: samtools version 0.1.19 is up to date; not installing
INFO: Custom install for 'samtools' end time: 2013-12-11 19:42:51.047238;
duration: 0:00:00.095976
INFO: Custom install for 'tabix' start time: 2013-12-11 19:42:51.047637
DBG [fabfile.py]: Importing module cloudbio.custom.bio_nextgen
DBG [fabfile.py]: Looking for custom install function
cloudbio.custom.bio_nextgen.install_tabix
INFO: Custom install for 'tabix' end time: 2013-12-11 19:42:51.112410;
duration: 0:00:00.064773
INFO: Config end time: 2013-12-11 19:42:51.113703; duration: 0:00:00.567177

Done.
Installing gemini data...
Checking required dependencies...
Traceback (most recent call last):
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/install-data.py",
line 143, in
install_annotation_files(anno_dir)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/install-data.py",
line 72, in install_annotation_files
_check_dependencies()
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/install-data.py",
line 84, in _check_dependencies
subprocess.check_call([cmd, "--version"], stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/subprocess.py",
line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['curl', '--version']' returned
non-zero exit status 2
Traceback (most recent call last):
File "gemini_install.py", line 254, in
main(parser.parse_args())
File "gemini_install.py", line 54, in main
install_data(gemini["python"], gemini["data_script"], args.datadir)
File "gemini_install.py", line 164, in install_data
subprocess.check_call([python_cmd, data_script, datadir])
File
"/opt/az/local/bcbio-nextgen/use_git_clone/installdir/anaconda/lib/python2.7/subprocess.py",
line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command
'['/opt/az/local/gemini/installdir/share/gemini/anaconda/bin/python',
'/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/install-data.py',
'/opt/az/local/gemini/installdir/share/gemini']' returned non-zero exit
status 1
[root@green gemini]#

On Wed, Dec 11, 2013 at 11:58 AM, Brad Chapman notifications@github.comwrote:

Jeff;
Unfortunately I don't have a better workaround for this one than trying
again to see if it's persistent. It looks like a one off network error
(which could also be due to pypi). pybedtools is also used inside
bcbio-nextgen so this "should work" on the network since it had to be done
once during the original bcbio-nextgen install. I hope retry magic will
work. Crossing fingers.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-30338915
.

@chapmanb

This comment has been minimized.

Member

chapmanb commented Dec 11, 2013

Jeff;
Apologies, you're right on there is another check in the code that installs the data. I pushed fixes for this to gemini so since you're so far through the process you should be able to restart with:

gemini update --devel

and it will grab the fixed code and then kick off the data downloads again. There is a bunch of data to install but you're at the very last step so hopefully this pushes things through. Thanks again for the patience.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 11, 2013

Hi Brad,

Man I hate to keep bugging you. Still not quite there. This is what I get
now:

[root@green gemini]# /opt/az/local/gemini/installdir/bin/gemini update
--devel
Searching for distribute
Reading https://pypi.python.org/simple/distribute/
Download error on https://pypi.python.org/simple/distribute/: [Errno 1]
_ssl.c:507: error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some
packages may not be found!
Couldn't find index page for 'distribute' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: [Errno 1] _ssl.c:507:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify
failed -- Some packages may not be found!
No local packages or download links found for distribute
error: Could not find suitable distribution for
Requirement.parse('distribute')
Downloading/unpacking distribute
Downloading distribute-0.7.3.zip (145kB): 145kB downloaded
Running setup.py egg_info for package distribute

Downloading/unpacking setuptools>=0.7 from
https://pypi.python.org/packages/source/s/setuptools/setuptools-2.0.tar.gz#md5=887cdf417bc5d25326c66bb98c0dd160(from
distribute)
Downloading setuptools-2.0.tar.gz (783kB): 783kB downloaded
Running setup.py egg_info for package setuptools

Installing collected packages: setuptools, distribute
Found existing installation: setuptools 1.4
Cleaning up...
Cannot remove entries from nonexistent file
/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/easy-install.pth
Storing complete log in /root/.pip/pip.log
Traceback (most recent call last):
File "/opt/az/local/gemini/installdir/bin/gemini", line 5, in
gemini.gemini_main.main()
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py",
line 786, in main
args.func(parser, args)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_update.py",
line 24, in release
subprocess.check_call([pip_bin, "install", "--upgrade", "distribute"])
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/subprocess.py",
line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command
'['/opt/az/local/gemini/installdir/share/gemini/anaconda/bin/pip',
'install', '--upgrade', 'distribute']' returned non-zero exit status 1
[root@green gemini]#

On Wed, Dec 11, 2013 at 3:43 PM, Brad Chapman notifications@github.comwrote:

Jeff;
Apologies, you're right on there is another check in the code that
installs the data. I pushed fixes for this to gemini so since you're so far
through the process you should be able to restart with:

gemini update --devel

and it will grab the fixed code and then kick off the data downloads
again. There is a bunch of data to install but you're at the very last step
so hopefully this pushes things through. Thanks again for the patience.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-30361759
.

@chapmanb

This comment has been minimized.

Member

chapmanb commented Dec 12, 2013

Jeff;
Thanks so much for the patience with this. That's the same error you ran into on the installer. I'd fixed it in the update code but unfortunately it's a bit of a chicken and an egg problem since you need the installed update code to work before doing the update. We can work around that if you manually grab the updated file, then rerun the update:

wget -O  /opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_update.py https://raw.github.com/arq5x/gemini/master/gemini/gemini_update.py
gemini update --devel

Fingers crossed this will sort it all out. Sorry again about all the issues here and thanks for the help resolving them.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 12, 2013

Hi Brad,

I believe we're almost there. I now get:

changing mode of

/opt/az/local/gemini/installdir/share/gemini/anaconda/bin/gemini to 755
Successfully installed gemini
Cleaning up...
Traceback (most recent call last):
File "/opt/az/local/gemini/installdir/bin/gemini", line 5, in
gemini.gemini_main.main()
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py",
line 786, in main
args.func(parser, args)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_update.py",
line 44, in release
config = gemini.config.read_gemini_config()
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/config.py",
line 46, in read_gemini_config
fname = _get_config_file(dirs)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/config.py",
line 42, in _get_config_file
.format(CONFIG_FILE, dnames))
ValueError: GEMINI configuration file gemini-config.yaml not found in
['/opt/az/local/gemini/installdir/share/gemini/',
'/opt/az/local/gemini/installdir/share/gemini/gemini',
'/usr/local/share/gemini', '/root/.gemini'].
Please ensure the GEMINI data is installed using the install-data.py script
http://gemini.readthedocs.org/en/latest/content/installation.html

I could not find a file called 'gemini-config.yaml', but I found 3
'install-data.py' files. The 2 under the 'gemini-master' directory are
identical, but slightly different from the one under the installdir
directory, which is newer.

[root@green gemini]# pwd
/opt/az/local/gemini
[root@green gemini]# find . -name gemini-config.yaml -print
[root@green gemini]# find . -name install-data.py -print
./installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/install-data.py
./gemini-master/gemini/install-data.py
./gemini-master/build/lib/gemini/install-data.py

I attempted to run this using:

[root@green gemini]# python
installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/install-data.py
/opt/az/local/gemini/installdir/share
Traceback (most recent call last):
File
"installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/install-data.py",
line 8, in
from gemini.config import read_gemini_config, write_gemini_config
ImportError: No module named gemini.config
[root@green gemini]#

To be honest, I'm not sure if the final value
'/opt/az/local/gemini/installdir/share' is the proper directory to use.
However, the current problem is there is no gemini.config file anywhere.

[root@green gemini]# pwd
/opt/az/local/gemini
[root@green gemini]# find . -name gemini.config -print
[root@green gemini]#

Any thoughts what my next step should be?

Jeff

On Thu, Dec 12, 2013 at 4:37 AM, Brad Chapman notifications@github.comwrote:

Jeff;
Thanks so much for the patience with this. That's the same error you ran
into on the installer. I'd fixed it in the update code but unfortunately
it's a bit of a chicken and an egg problem since you need the installed
update code to work before doing the update. We can work around that if you
manually grab the updated file, then rerun the update:

wget -O /opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_update.py https://raw.github.com/arq5x/gemini/master/gemini/gemini_update.py
gemini update --devel

Fingers crossed this will sort it all out. Sorry again about all the
issues here and thanks for the help resolving them.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-30401674
.

@chapmanb

This comment has been minimized.

Member

chapmanb commented Dec 12, 2013

Jeff;
I am going to give up writing installers forever. Sorry to keep leaving you in terrible states. The good news is the configuration file is super easy to add and then you'll be good to go. If you installed gemini in /opt/az/local/gemini (you should have /opt/az/local/gemini/anaconda), then stick this into `/opt/az/local/gemini/gemini-config.yaml':

annotation_dir: /opt/az/local/gemini/data
versions:
  GRCh37-gms-mappability.vcf.gz: 2
  hg19.rmsk.bed.gz: 2

then rerun the update command. Fingers crossed this gets you to downloading. Let me know if anything else at all crops up and thanks for the patience.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 12, 2013

Hi Brad,

Still not quite there. I include everything below. Could the errors near
the top of this be clues, e.g. the download error leading to 'some packages
may not be found'?

This is what I have:

[root@green gemini]# pwd
/opt/az/local/gemini
[root@green gemini]# ls
bcbio_nextgen_install.py gemini_install.py.bad.2 installdir
gemini-config.yaml gemini_install.py.bad.3 tmpgemini_install
gemini_install.py gemini-master
gemini_install.py.bad gemini-master.zip
[root@green gemini]#

This directory (/opt/az/local/gemini) is where I created
gemini-config.yaml, which looks like:

[root@green gemini]# more gemini-config.yaml
annotation_dir: /opt/az/local/gemini/data
versions:
GRCh37-gms-mappability.vcf.gz: 2
hg19.rmsk.bed.gz: 2
[root@green gemini]#

Now here's the command I gave from the /opt/az/local/gemini directory,
followed by the full output:

[root@green gemini]# /opt/az/local/gemini/installdir/bin/gemini update
--devel
Searching for distribute
Reading https://pypi.python.org/simple/distribute/
Download error on https://pypi.python.org/simple/distribute/: [Errno 1]
_ssl.c:507: error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some
packages may not be found!
Couldn't find index page for 'distribute' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: [Errno 1] _ssl.c:507:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify
failed -- Some packages may not be found!
No local packages or download links found for distribute
error: Could not find suitable distribution for
Requirement.parse('distribute')
Downloading/unpacking distribute
Downloading distribute-0.7.3.zip (145kB): 145kB downloaded
Running setup.py egg_info for package distribute

Downloading/unpacking setuptools>=0.7 from
https://pypi.python.org/packages/source/s/setuptools/setuptools-2.0.tar.gz#md5=887cdf417bc5d25326c66bb98c0dd160(from
distribute)
Downloading setuptools-2.0.tar.gz (783kB): 783kB downloaded
Running setup.py egg_info for package setuptools

Installing collected packages: setuptools, distribute
Found existing installation: setuptools 1.4
Cleaning up...
Cannot remove entries from nonexistent file
/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/easy-install.pth
Storing complete log in /root/.pip/pip.log

All requested packages already installed.

packages in environment at

/opt/az/local/gemini/installdir/share/gemini/anaconda:

cython 0.19.2 py27_0
ipython 1.1.0 py27_0
jinja2 2.7.1 py27_0
nose 1.3.0 py27_0
numpy 1.7.1 py27_2
pandas 0.12.0 np17py27_0
pip 1.4.1 py27_1
pycrypto 2.6.1 py27_0
pyparsing 1.5.6 py27_0
pysam 0.6 py27_0
pyyaml 3.10 py27_0
pyzmq 2.2.0.1 py27_1
scipy 0.13.1 np17py27_1
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.1 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade):
pyparsing>=1.5.6,<=1.5.7 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): pysam>=0.6 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): cyvcf>=0.1.9.1 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): PyYAML>=3.10 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): pybedtools>=0.6.2
in ./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7.1 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade):
python-graph-core>=1.8.2 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 8))
Requirement already satisfied (use --upgrade to upgrade):
python-graph-dot>=1.8.2 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): bottle>=0.11.6 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 10))
Requirement already satisfied (use --upgrade to upgrade):
ipython-cluster-helper>=0.1.9 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): bx-python>=0.7.1
in ./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): pandas>=0.11.0 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 13))
Requirement already satisfied (use --upgrade to upgrade): scipy>=0.12.0 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 14))
Downloading/unpacking gemini==0.6.3.2 (from -r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 15))
Downloading gemini-0.6.3.2.tar.gz (10.2MB): 10.2MB downloaded
Running setup.py egg_info for package gemini

Requirement already satisfied (use --upgrade to upgrade): pydot in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from
python-graph-dot>=1.8.2->-r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): pyzmq>=2.1.11 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from
ipython-cluster-helper>=0.1.9->-r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): ipython>=1.1.0 in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages (from
ipython-cluster-helper>=0.1.9->-r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): setuptools in
./installdir/share/gemini/anaconda/lib/python2.7/site-packages/setuptools-1.4-py2.7.egg
(from pydot->python-graph-dot>=1.8.2->-r
https://raw.github.com/arq5x/gemini/master/requirements.txt (line 9))
Installing collected packages: gemini
Found existing installation: gemini 0.6.4a
Uninstalling gemini:
Successfully uninstalled gemini
Running setup.py install for gemini

changing mode of build/scripts-2.7/gemini from 644 to 755
changing mode of

/opt/az/local/gemini/installdir/share/gemini/anaconda/bin/gemini to 755
Successfully installed gemini
Cleaning up...
Installing latest GEMINI development version
Downloading/unpacking git+https://github.com/arq5x/gemini.git
Cloning https://github.com/arq5x/gemini.git to /tmp/pip-CijquS-build
Running setup.py egg_info for package from git+
https://github.com/arq5x/gemini.git

Installing collected packages: gemini
Found existing installation: gemini 0.6.3.2
Uninstalling gemini:
Successfully uninstalled gemini
Running setup.py install for gemini

changing mode of build/scripts-2.7/gemini from 644 to 755
changing mode of

/opt/az/local/gemini/installdir/share/gemini/anaconda/bin/gemini to 755
Successfully installed gemini
Cleaning up...
Traceback (most recent call last):
File "/opt/az/local/gemini/installdir/bin/gemini", line 5, in
gemini.gemini_main.main()
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py",
line 786, in main
args.func(parser, args)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_update.py",
line 44, in release
config = gemini.config.read_gemini_config()
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/config.py",
line 46, in read_gemini_config
fname = _get_config_file(dirs)
File
"/opt/az/local/gemini/installdir/share/gemini/anaconda/lib/python2.7/site-packages/gemini/config.py",
line 42, in _get_config_file
.format(CONFIG_FILE, dnames))
ValueError: GEMINI configuration file gemini-config.yaml not found in
['/opt/az/local/gemini/installdir/share/gemini/',
'/opt/az/local/gemini/installdir/share/gemini/gemini',
'/usr/local/share/gemini', '/root/.gemini'].
Please ensure the GEMINI data is installed using the install-data.py script
http://gemini.readthedocs.org/en/latest/content/installation.html
[root@green gemini]#

On Thu, Dec 12, 2013 at 10:56 AM, Brad Chapman notifications@github.comwrote:

Jeff;
I am going to give up writing installers forever. Sorry to keep leaving
you in terrible states. The good news is the configuration file is super
easy to add and then you'll be good to go. If you installed gemini in
/opt/az/local/gemini (you should have /opt/az/local/gemini/anaconda),
then stick this into `/opt/az/local/gemini/gemini-config.yaml':

annotation_dir: /opt/az/local/gemini/data
versions:
GRCh37-gms-mappability.vcf.gz: 2
hg19.rmsk.bed.gz: 2

then rerun the update command. Fingers crossed this gets you to
downloading. Let me know if anything else at all crops up and thanks for
the patience.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-30434531
.

@chapmanb

This comment has been minimized.

Member

chapmanb commented Dec 12, 2013

Jeff;
Based on that traceback it looks like you've installed gemini in /opt/az/local/gemini/installdir/share/gemini/, so you want /opt/az/local/gemini/installdir/share/gemini/gemini-config.yaml as:

annotation_dir:  /opt/az/local/gemini/installdir/share/gemini/data 
versions: 
  GRCh37-gms-mappability.vcf.gz: 2
  hg19.rmsk.bed.gz: 2

Sorry, I think I misinterpreted where you were installing gemini. Apologies also for all this manual nonsense. Hope this does it.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Dec 12, 2013

SUCCESS!!!

The first time I tried this last attempt it died. Then I looked at the
messages. Timed out. Changed my proxy yet again (have I told you yet how
much I despise our network?) and tried again. It worked. Here's the end
of all the downloads:

  • downloading
    https://s3.amazonaws.com/gemini-annotations/cse-hiseq-8_4-2013-02-20.bed.gzto
    /opt/az/local/gemini/data

    % Total % Received % Xferd Average Speed Time Time Time
    Current
    Dload Upload Total Spent Left
    Speed
    100 24.7M 100 24.7M 0 0 2566k 0 0:00:09 0:00:09 --:--:--
    3030k

  • downloading
    https://s3.amazonaws.com/gemini-annotations/cse-hiseq-8_4-2013-02-20.bed.gz.tbito
    /opt/az/local/gemini/data

    % Total % Received % Xferd Average Speed Time Time Time
    Current
    Dload Upload Total Spent Left
    Speed
    100 1568k 100 1568k 0 0 1931k 0 --:--:-- --:--:-- --:--:--
    2605k
    Gemini upgraded to latest version
    Cloning into 'gemini'...
    remote: Counting objects: 4534, done.
    remote: Compressing objects: 100% (1701/1701), done.
    remote: Total 4534 (delta 2975), reused 4339 (delta 2788)
    Receiving objects: 100% (4534/4534), 32.24 MiB | 629.00 KiB/s, done.
    Resolving deltas: 100% (2975/2975), done.
    Checking connectivity... done.
    Run test suite with: cd /opt/az/local/gemini/installdir/share/gemini/gemini
    && bash master-test.sh
    [root@green gemini]#

Now we'll see if we can get it to run.

I was just told that our agreement with you guys has to do with
bcbio-nextgen, but nothing to do with gemini. So apparently you really
didn't have to help with this. Thanks for all the help I appreciate it.
I've been trying to work on 4 different top priority projects at the same
time. (Really.)

Jeff

On Thu, Dec 12, 2013 at 3:40 PM, Brad Chapman notifications@github.comwrote:

Jeff;
Based on that traceback it looks like you've installed gemini in
/opt/az/local/gemini/installdir/share/gemini/, so you want
/opt/az/local/gemini/installdir/share/gemini/gemini-config.yaml as:

annotation_dir: /opt/az/local/gemini/installdir/share/gemini/data
versions:
GRCh37-gms-mappability.vcf.gz: 2
hg19.rmsk.bed.gz: 2

Sorry, I think I misinterpreted where you were installing gemini.
Apologies also for all this manual nonsense. Hope this does it.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-30459607
.

@chapmanb

This comment has been minimized.

Member

chapmanb commented Dec 13, 2013

Jeff;
Brilliant, glad it's working and sorry about all the required back and forth. This helped on my side to make the installer more bullet proof so hopefully it will be easier in the future. Thank you again for the patience.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Jan 16, 2014

Hi Brad,

I'm trying to install the latest version, 0.7.6. I'm using:

python bcbio_nextgen_install.py
/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir
--tooldir=/opt/az/local/bcbio-nextgen/stable/0.7.6/tooldir --nodata

However I'm eventually getting

Upgrading third party tools to latest versions
Cloning into 'cloudbiolinux'...
error: couldn't connect to host while accessing
https://github.com/chapmanb/cloudbiolinux.git/info/refs
fatal: HTTP request failed
Traceback (most recent call last):
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/bin/bcbio_nextgen.py",
line 43, in
install.upgrade_bcbio(kwargs["args"])
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/site-packages/bcbio/install.py",
line 51, in upgrade_bcbio
upgrade_thirdparty_tools(args, REMOTES)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/site-packages/bcbio/install.py",
line 185, in upgrade_thirdparty_tools
cbl = get_cloudbiolinux(remotes)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/site-packages/bcbio/install.py",
line 326, in get_cloudbiolinux
subprocess.check_call(["git", "clone", remotes["cloudbiolinux"]])
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/subprocess.py",
line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'clone', '
https://github.com/chapmanb/cloudbiolinux.git']' returned non-zero exit
status 128
Traceback (most recent call last):
File "bcbio_nextgen_install.py", line 232, in
main(parser.parse_args(), sys.argv[1:])
File "bcbio_nextgen_install.py", line 37, in main
subprocess.check_call([bcbio["bcbio_nextgen.py"], "upgrade"] +
_clean_args(sys_argv, args, bcbio))
File
"/opt/az/local/bcbio-nextgen/use_git_clone/installdir/anaconda/lib/python2.7/subprocess.py",
line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command
'['/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/bin/bcbio_nextgen.py',
'upgrade', '--tooldir=/opt/az/local/bcbio-nextgen/stable/0.7.6/tooldir']'
returned non-zero exit status 1

I"ve tried with various proxies here, so I don't think it's a problem on
our end. Can you check if there's a problem at the other end?

Thanks.

Jeff

On Thu, Dec 12, 2013 at 8:44 PM, Brad Chapman notifications@github.comwrote:

Jeff;
Brilliant, glad it's working and sorry about all the required back and
forth. This helped on my side to make the installer more bullet proof so
hopefully it will be easier in the future. Thank you again for the patience.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-30480002
.

@roryk

This comment has been minimized.

Collaborator

roryk commented Jan 16, 2014

Hi Jeff,

You should be able to just update the existing installation instead of reinstalling the whole thing:

bcbio_nextgen.py upgrade
@chapmanb

This comment has been minimized.

Member

chapmanb commented Jan 16, 2014

Jeff;
I can confirm that cloning cloudbiolinux from GitHub works cleanly. It is trying to run:

git clone https://github.com/chapmanb/cloudbiolinux.git

so you can test from the command line separate from the installer to see if that goes cleanly. I don't know if we ever got a good resolution for this issue previously; it seemed to be network issues that went away.

@mjafin

This comment has been minimized.

Contributor

mjafin commented Jan 16, 2014

(We're keeping separate versions in separate installs, that's why installing from scratch)

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Jan 16, 2014

I get:

[root@green 0.7.6]# git clone https://github.com/chapmanb/cloudbiolinux.git
Cloning into 'cloudbiolinux'...
error: couldn't connect to host while accessing
https://github.com/chapmanb/cloudbiolinux.git/info/refs
fatal: HTTP request failed

Maybe we're having proxy problems here, though I've tested with multiple
proxies in multiple locations. Let me look thru our old emails to see if
there's anything we came up with before.

Jeff

On Thu, Jan 16, 2014 at 11:34 AM, Brad Chapman notifications@github.comwrote:

Jeff;
I can confirm that cloning cloudbiolinux from GitHub works cleanly. It is
trying to run:

git clone https://github.com/chapmanb/cloudbiolinux.git

so you can test from the command line separate from the installer to see
if that goes cleanly. I don't know if we ever got a good resolution for
this issue previously; it seemed to be network issues that went away.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-32485541
.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Jan 16, 2014

I tried a different proxy that Miika suggested. Still a problem, but
different. I now get:

[root@green 0.7.6]# python bcbio_nextgen_install.py
/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir
--tooldir=/opt/az/local/bcbio-nextgen/stable/0.7.6/tooldir --nodata
Checking required dependencies
git version 1.7.8.4
Installing isolated base python installation
Installing bcbio-nextgen
An unexpected error has occurred, please consider sending the
following traceback to the conda GitHub issue tracker at:

https://github.com/pydata/conda/issues

Include the output of the command 'conda info' in your report.

Traceback (most recent call last):
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/bin/conda",
line 5, in
sys.exit(main())
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/site-packages/conda/cli/main.py",
line 186, in main
args.func(args, p)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/site-packages/conda/cli/main_install.py",
line 161, in execute
args.override_channels)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/site-packages/conda/api.py",
line 33, in get_index
return fetch_index(tuple(channel_urls))
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/site-packages/conda/utils.py",
line 61, in call
value = self.func(_args)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/site-packages/conda/fetch.py",
line 108, in fetch_index
repodata = fetch_repodata(url)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/site-packages/conda/fetch.py",
line 72, in fetch_repodata
u = connectionhandled_urlopen(request)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/site-packages/conda/connection.py",
line 83, in connectionhandled_urlopen
return urllib2.urlopen(request)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/urllib2.py",
line 127, in urlopen
return _opener.open(url, data, timeout)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/urllib2.py",
line 404, in open
response = self._open(req, data)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/urllib2.py",
line 422, in _open
'_open', req)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/urllib2.py",
line 382, in _call_chain
result = func(_args)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/urllib2.py",
line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/urllib2.py",
line 1187, in do_open
r = h.getresponse(buffering=True)
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/httplib.py",
line 1045, in getresponse
response.begin()
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/httplib.py",
line 409, in begin
version, status, reason = self._read_status()
File
"/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/lib/python2.7/httplib.py",
line 373, in _read_status
raise BadStatusLine(line)
httplib.BadStatusLine: ''
Traceback (most recent call last):
File "bcbio_nextgen_install.py", line 232, in
main(parser.parse_args(), sys.argv[1:])
File "bcbio_nextgen_install.py", line 34, in main
install_conda_pkgs(anaconda)
File "bcbio_nextgen_install.py", line 90, in install_conda_pkgs
subprocess.check_call([anaconda["conda"], "install", "--yes"] + pkgs)
File
"/opt/az/local/bcbio-nextgen/use_git_clone/installdir/anaconda/lib/python2.7/subprocess.py",
line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command
'['/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/bin/conda',
'install', '--yes', 'biopython', 'boto', 'cython', 'ipython', 'lxml',
'matplotlib', 'nose', 'numpy', 'pandas', 'patsy', 'pycrypto', 'pip',
'pysam', 'pyyaml', 'pyzmq', 'requests', 'scipy', 'tornado', 'statsmodels']'
returned non-zero exit status 1

Here's the output from 'conda info'

[root@green 0.7.6]# conda info
Current conda install:

         platform : linux-64
    conda version : 2.2.8
 root environment :

/opt/az/local/bcbio-nextgen/use_git_clone/installdir/anaconda (writable)
default environment :
/opt/az/local/bcbio-nextgen/use_git_clone/installdir/anaconda
envs directories :
/opt/az/local/bcbio-nextgen/use_git_clone/installdir/anaconda/envs
package cache :
/opt/az/local/bcbio-nextgen/use_git_clone/installdir/anaconda/pkgs
channel URLs : http://repo.continuum.io/pkgs/free/linux-64/
http://repo.continuum.io/pkgs/pro/linux-64/
config file : None
is foreign system : False

[root@green 0.7.6]#

On Thu, Jan 16, 2014 at 11:34 AM, Brad Chapman notifications@github.comwrote:

Jeff;
I can confirm that cloning cloudbiolinux from GitHub works cleanly. It is
trying to run:

git clone https://github.com/chapmanb/cloudbiolinux.git

so you can test from the command line separate from the installer to see
if that goes cleanly. I don't know if we ever got a good resolution for
this issue previously; it seemed to be network issues that went away.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-32485541
.

@chapmanb

This comment has been minimized.

Member

chapmanb commented Jan 16, 2014

Jeff;
It looks like something is now messed up with the proxy and conda, the python package manager. This should be a reproducible command line argument to test:

/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/bin/conda install biopython

I believe it uses all_proxy, http_proxy and https_proxy but don't know enough to have specific suggestions. Sorry, hope this helps some.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Jan 16, 2014

From the above command I get:

[root@green 0.7.6]#
/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/bin/conda
install biopython

All requested packages already installed.

packages in environment at

/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda:

biopython 1.63 np17py27_0
[root@green 0.7.6]#

On Thu, Jan 16, 2014 at 2:24 PM, Brad Chapman notifications@github.comwrote:

Jeff;
It looks like something is now messed up with the proxy and conda, the
python package manager. This should be a reproducible command line argument
to test:

/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/bin/conda install biopython

I believe it uses all_proxy, http_proxy and https_proxy but don't know
enough to have specific suggestions. Sorry, hope this helps some.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-32520144
.

@chapmanb

This comment has been minimized.

Member

chapmanb commented Jan 16, 2014

Jeff;
Well that's good, it looks like conda installs aren't totally broken since some parts got installed. If you re-run the install does it fail consistently at the same place, or was it a one-off network error?

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Jan 16, 2014

Continues to fail at the same place.

On Thu, Jan 16, 2014 at 3:23 PM, Brad Chapman notifications@github.comwrote:

Jeff;
Well that's good, it looks like conda installs aren't totally broken since
some parts got installed. If you re-run the install does it fail
consistently at the same place, or was it a one-off network error?


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-32537297
.

@chapmanb

This comment has been minimized.

Member

chapmanb commented Jan 16, 2014

Jeff;
I guess the next step is to find the reproducible failure. You can do:

/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/bin/conda list

to get the packages that actually got installed and compare against what it is trying to install:

[ 'biopython', 'boto', 'cython', 'ipython', 'lxml', 'matplotlib', 'nose', 'numpy', 'pandas', 'patsy', 'pycrypto', 'pip', 'pysam', 'pyyaml', 'pyzmq', 'requests', 'scipy', 'tornado', 'statsmodels']

to guess what it is dying on.

@saxejeffreyd

This comment has been minimized.

saxejeffreyd commented Jan 16, 2014

Okay, I think I got it. The .gitconfig file had a proxy specified
(actually the one that's usually the best). When I changed that to a
different proxy it worked. So yet again, our miserable proxies here.

Jeff

On Thu, Jan 16, 2014 at 3:37 PM, Brad Chapman notifications@github.comwrote:

Jeff;
I guess the next step is to find the reproducible failure. You can do:

/opt/az/local/bcbio-nextgen/stable/0.7.6/installdir/anaconda/bin/conda list

to get the packages that actually got installed and compare against what
it is trying to install:

[ 'biopython', 'boto', 'cython', 'ipython', 'lxml', 'matplotlib', 'nose', 'numpy', 'pandas', 'patsy', 'pycrypto', 'pip', 'pysam', 'pyyaml', 'pyzmq', 'requests', 'scipy', 'tornado', 'statsmodels']

to guess what it is dying on.


Reply to this email directly or view it on GitHubhttps://github.com//issues/207#issuecomment-32544378
.

czarifis pushed a commit to czarifis/gemini that referenced this issue Jul 21, 2014

czarifis pushed a commit to czarifis/gemini that referenced this issue Jul 21, 2014

@zytwh2012

This comment has been minimized.

zytwh2012 commented May 15, 2018

git config --global http.postBuffer 524288000 works for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment