Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Access Swestore National Storage via the iRODS SRM Module #1

Closed
samuell opened this issue Nov 11, 2011 · 18 comments
Closed

Access Swestore National Storage via the iRODS SRM Module #1

samuell opened this issue Nov 11, 2011 · 18 comments
Assignees
@jhagberg
Milestone
iRODS Swestore ac...

Comments

@samuell
Copy link
Contributor

samuell commented Nov 11, 2011

Compile and install the iRODS SRM module available at:
http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/grid_computing/iRODS
... and use it to connect to SweStore National Storage.
@ghost ghost assigned samuell Nov 11, 2011
@ghost ghost assigned jhagberg Dec 13, 2011
@samuell
Copy link
Contributor Author

samuell commented Dec 13, 2011

@jhagberg: I'm realizing more and more how hard it is for me to find time to work on these things ... Do you think you could have a look on this one, as soon as we get up a test node, where you have sudo access?

Feel free to tell if you think we should split our work in another way. I just think this is kind of the main stumbling block right now before we can start to play with things in practice, thus it would be nice to put some focus on getting this fixed first/asap ...

@samuell
Copy link
Contributor Author

samuell commented Dec 13, 2011

@jhagberg: ... I will see if I find some time to get started on this though, but feel free to dig in if you feel like it! :) ... the sooner the better ...

@jhagberg
Copy link
Contributor

To work with this we need a new server grid certificate.

Perhaps we need a new test volume on swestore and a server proxy cert for u5.
Samuel can you fix this?

@jhagberg
Copy link
Contributor

@samuell Can you also check with Swestore if it is possible to use gsiftp?

Does it exist a
gsiftp://swestore.se?

or is arccp the only way?

@samuell
Copy link
Contributor Author

samuell commented Dec 14, 2011

@jhagberg: You are very much right about a test volume. Always thought it feels scary to play with the full uppnex account ... especially since there is not even backup on Swestore, only simple mirroring.

Regarding GsiFTP, I have been talking to the Swestore guys about this earlier. They say that the SRM protocol (which is used by the ng* commands) actually wraps the GsiFTP, which in turn is told to be some kind of subset of GridFTP.

But, further down on the GridFTP module page (http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/grid_computing/iRODS) , they actually have an "SRM" module too, so my thought was to try that module directly!

@samuell
Copy link
Contributor Author

samuell commented Dec 14, 2011

(Have now mailed Tom and Swestore Support, for a test volume, and server cert for u5 ...)

@jhagberg
Copy link
Contributor

Yes but it say that the SRM with gsftp module make it possible to load
files on and off dCACHE
Den 14 dec 2011 17:32 skrev "Samuel Lampa" <
reply@reply.github.com

:

@jhagberg: You are very much right about a test volume. Always thought it
feels scary to play with the full uppnex account ... especially since there
is not even backup on Swestore, only simple mirroring.

Regardgin GsiFTP, I have been talking to the Swestore guys about this
earlier. They say that the SRM protocol (which is used by the ng* commands)
actually wraps the GsiFTP, which in turn is told to be some kind of subset
of GridFTP.

But, further down on the GridFTP module page (
http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/grid_computing/iRODS)
, they actually have an "SRM" module too, so my thought was to try that
module directly!

Reply to this email directly or view it on GitHub:
#1 (comment)

@jhagberg
Copy link
Contributor

Server cert you need to fix yorself for the server.
Den 14 dec 2011 17:40 skrev "Samuel Lampa" <
reply@reply.github.com

:

(Have now mailed Tom and Swestore Support, for a test volume, and server
cert for u5 ...)

Reply to this email directly or view it on GitHub:
#1 (comment)

@samuell
Copy link
Contributor Author

samuell commented Dec 14, 2011

@jhagberg Ah, right .... but, "GridFTP" you mean? (

"[...] Combinded with the GridFTP module SRM based file transfers are possible [...]"
http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/grid_computing/iRODS

Well, I'll raise the question with the SweStore guys again!

@jhagberg
Copy link
Contributor

Yes and in the module readme they use gsiftp://
Den 14 dec 2011 17:46 skrev "Samuel Lampa" <
reply@reply.github.com

:

@jhagberg Ah, right .... but, "GridFTP" you mean? (

"[...] Combinded with the GridFTP module SRM based file transfers are
possible [...]"

http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/grid_computing/iRODS

Well, I'll raise the question with the SweStore guys again!

Reply to this email directly or view it on GitHub:
#1 (comment)

@samuell
Copy link
Contributor Author

samuell commented Dec 14, 2011

Ah, I see!

Mattias Wadenstein that there is gsiftp addresses, but no permanent ones ... instead one has to talk the SRM protocol in order to get a GsiFTP dynamically. (I guess this is part of how they do the "round-robin" functionality of spreading things on different storage pools ...).

Excerpt from the chat:

----------Swestore chat-room-----------
samuel.lampa 05:47:26 PM
Hmm, alltså, jag har frågat förr, men vad var skillnaden mellan GsiFTP och GridFTP? ... borde en mjukvara som stöder SRM+GridFTP funka bra för att ansluta till nationella lagringen?
(... även om den ju sägs ha SRM+GsiFTP?)
05:50:57 PM
Har förstått att den ena ska vara ett subset av den andra, men har inte riktigt kläm på vad det innebär ...
05:51:26 PM

samuel.lampa 06:02:49 PM
Aha, vid närmare koll så använder modulen till och med "gsiftp/" i URL:en, så uppenbarligen borde det fungera, i teorin.

maswan 06:02:55 PM
gridftp och gsiftp är samma sak

samuel.lampa 06:03:00 PM
Ok!
Då blir frågan om man kan ansluta direkt med gsiftp på nått sätt (finns det fasta gsiftp URL:er), eller måste man få dem via SRM?
06:03:43 PM

maswan 06:04:19 PM
det finns, ja. men de är inte uppsatta med några permanenta namn eller ens namespace

samuel.lampa 06:04:52 PM
Ok, hmm ... men om man förstår hur man pratar SRM så får man en dynamiskt?

maswan 06:04:59 PM
precis

samuel.lampa 06:05:36 PM
Ok, vi får kika närmare på hur mjukvaran ifråga (SRM modul för iRODS) fungerar då. Danke för feedbacken!

maswan 06:07:09 PM
om du har något som bara funkar med gridftp och du vill göra rena mjukvarutester så kan du ju ta den gridftp-dörr som du får i ett srm-svar (arccp med lämplig debugflagga etc), men det är inget som är uppsatt att köras så permanent.
om ni sedan kommer fram till att ni behöver en gridftp-dörr med stabil URL etc så får du nog förhandla med tom/jens/nikke/foo

samuel.lampa 06:09:51 PM
Ja, Ok!
----------End: Swestore chat-room-----------

@samuell
Copy link
Contributor Author

samuell commented Dec 16, 2011

Have applied for a TCS server cert for u5 now.

Also, Tom said that we can create a testfolder under the the "ops" folder, if we become member of (or let the server cert become a member of) the ops_test voms group.

Just gotta wait for the cert paper work to get through ...

@jhagberg
Copy link
Contributor

@samuell Could you check if "Globus GSI Libraries" are on place on u5 and Globus Toolkit libraries
I am missing srm_types.h when trying to complie srm module

@jhagberg
Copy link
Contributor

Hej
Need to have
srm-ifce installed

I do not know yet if it is bundled in any rpm but it looks like it is a part of lcgutil
https://svnweb.cern.ch/trac/lcgutil/browser/srm-ifce?rev=4651

@samuell
Copy link
Contributor Author

samuell commented Dec 17, 2011

Ah, there we go ... bit closer to the solution ... will have a look ...

@samuell
Copy link
Contributor Author

samuell commented Dec 17, 2011

Aarrgh ... seem to only be rpms available for sl5 ... :(
http://etics-repository.cern.ch/repository/download/registered/emi/emi.lcgutil.srm-ifce/1.1.0

(Otherwise multiple srm stuff here: http://etics-repository.cern.ch/repository/download/registered/emi )

@jhagberg
Copy link
Contributor

Should we close this case?

@samuell
Copy link
Contributor Author

samuell commented Jan 11, 2012

@jhagberg I guess, yes, focusing on lftp/univMSSdriver instead, I guess ... (closing now)

@samuell samuell closed this as completed Jan 11, 2012
samuell pushed a commit that referenced this issue Jun 13, 2012
@trel
this is a test commit - no functional changes - testing github's issu…
6b326f1 
…e auto-linking - should link to #1

git-svn-id: svn://irodssvn.ucsd.edu/trunk@4956 ea137bd6-aa86-47b6-b90d-b101d9f19a90
samuell pushed a commit that referenced this issue Jun 13, 2012
@trel
linking worked - #1 is autolinked - very exciting
0d291d1 
see the full issue page here trel/irods#1





git-svn-id: svn://irodssvn.ucsd.edu/trunk@4957 ea137bd6-aa86-47b6-b90d-b101d9f19a90
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jhagberg jhagberg

Labels
None yet
Projects
None yet
Milestone
iRODS Swestore access in production
Development

No branches or pull requests
2 participants
@samuell @jhagberg