Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for rsync protocol #25

Closed
horkko opened this issue Nov 10, 2015 · 7 comments
Closed

Support for rsync protocol #25

horkko opened this issue Nov 10, 2015 · 7 comments
Assignees
@Chriou
Labels
enhancement
Milestone
3.1

Comments

@horkko
Copy link
Contributor

horkko commented Nov 10, 2015

Hi,

As of Java version, do you think rsync protocol could be supported in biomaj3?
Thanks

Emmanuel
@osallou
Copy link
Contributor

osallou commented Nov 10, 2015

I have to check for this, it was not really used. I do not know how it works to set regexp etc... for an rsync...

@osallou osallou added this to the 3.0.12 milestone Nov 10, 2015
@osallou osallou self-assigned this Nov 10, 2015
@horkko
Copy link
Contributor Author

horkko commented Aug 10, 2016

Hi,

Is there a chance to also have support for https?
I cannot get data from this site anymore: https://www.viprbrc.org/

Thanks

@osallou
Copy link
Contributor

osallou commented Aug 10, 2016

Https support should be straighforward, setting https protocol and in code
managing https along with http.
Strange we did not face issue before.

Could you open a different issue for this. A new developper on biomaj
should start on september, she could start by fixing this....

Le mer. 10 août 2016 12:19, Emmanuel Quevillon notifications@github.com a
écrit :

Hi,

Is there a chance to also have support for https?
I cannot get data from this site anymore: https://www.viprbrc.org/

Thanks

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#25 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA-gYuxIHiokWfaqP0rxBGMl-K-A8B3_ks5qeaW2gaJpZM4GfgTC
.

@osallou
Copy link
Contributor

osallou commented Aug 10, 2016

in the meanwhile , you can test with update in workflow.py:

    ...
    if protocol == 'http' or protocol == 'https:
        downloader = HTTPDownload(protocol, server, remote_dir, self.bank.config)
    if protocol == 'local':
        downloader = LocalDownload(remote_dir)
    if protocol == 'directftp':
        downloader = DirectFTPDownload('ftp', server, remote_dir, list_file)
    if protocol == 'directhttp':
        downloader = DirectHttpDownload('http', server, remote_dir, list_file)
    if protocol == 'directhttps':
        downloader = DirectHttpDownload('https', server, remote_dir, list_file)
    ...

@horkko
Copy link
Contributor Author

horkko commented Aug 10, 2016

Hi,

Investigating a little bit more, it seems, you're right, the protocol https should be straighforward.
I don't understand the problem I have.
I try to download a tarball which is located here:
https://www.viprbrc.org/brcDocs/datafiles/blast/DB_new_format/NONFLU_All.aa.tar.gz
Visible at this page:
https://www.viprbrc.org/brc/staticContent.spg?decorator=vipr&type=ViprInfo&subtype=Download

Here is my config:

protocol=directhttp
server=www.viprbrc.org/brcDocs/datafiles/blast/DB_new_format/
remote.dir=NONFLU_All.aa.tar.gz
target.name=%(remote.dir)s

It looks it finds it remotely as I have NONFLU_All.aa.tar.gz located in %(data.dir)/%(db.name)/<release>/flat
However, I cannot unarchive it as the file is not a tarball, but an html file as follow:

$ pwd
/data/db/biomaj/<db_name>
$ cat NONFLU_All.aa.tar.gz
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://www.viprbrc.org/brcDocs/datafiles/blast/DB_new_format///NONFLU_All.aa.tar.gz?">here</a>.</p>
</body></html>

If I do a wget on this file https://www.viprbrc.org/brcDocs/datafiles/blast/DB_new_format///NONFLU_All.aa.tar.gz the download works correctly. The same if I click the link in the corresponding pagehttps://www.viprbrc.org/brc/staticContent.spg?decorator=vipr&type=ViprInfo&subtype=Download

Do you have any clue? A misconfig I cannot see?

Thanks

Emmanuel

PS: Making code changes as requested, did not change anything

@osallou
Copy link
Contributor

osallou commented Aug 10, 2016

regarding changes, did you also updated he protocol in properties file to directhttps ?

It is a 302 Redirect from http to https. As you try to get it via http, this is "correct".
when https is handled, there will be no issue.

@osallou
Copy link
Contributor

osallou commented Aug 10, 2016

I have created an issue #55 for the HTTPs support

@horkko horkko mentioned this issue Aug 30, 2016
Open
@osallou osallou assigned Chriou and unassigned osallou Sep 8, 2016
@osallou osallou modified the milestones: 3.0.12, 3.1 Jan 2, 2017
osallou added a commit that referenced this issue Jan 20, 2017
@osallou
Upate feature/fix list for 3.1
2a931dd 
 Closes: #67,#66,#61,#74,#65,#25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Chriou Chriou

Labels
enhancement
Projects
None yet
Milestone
3.1
Development

No branches or pull requests
3 participants
@osallou @horkko @Chriou