AsyncStageout component

[DMWMBot]

Agent written in WMAgent component style and implements the main machinery of the AsyncStageout (the implementation of this machninery is described in the Ticket #95). It is needed to merge this agent in the AsyncStageout module.

[drsm79]

metson: Hassen, is this still valid? I think all these changes went in to #95 or am I wrong?

[DMWMBot]

riahi: Not all changes went to #95. The patches, which went in #95, were to have AsyncStageout as component. I aligned the patches of this ticket with changes done in ticket #95.

[drsm79]

metson: ok, cool thanks Hassen! I'll bring my local repo up to sync with your changes and hopefully commit tonight.

[DMWMBot]

riahi: Perfect!...Thank you

Hassen

[drsm79]

metson: This patch series looks good, and has been committed.

I have a few questions:

1. should TransferManager be removed? (I think yes)
2. is ''defaultProxy'' in the config the servers certificate? (I think it should be)
3. cleanSpace should probably reuse some code (not sure where from) to issue the appropriate delete command (IIRC lcg-rm is better in some cases) and/or deal with fallback/retries
4. getFTServer should read the mapping from a file specified in the configuration, instead of a hardcoded list
5. the orderFileAlgo should be replaced by changes to the CouchDB view, and how that's queried.

1 & 2 aren't a big deal (but I'd like to clarify before moving on). 3-5 should probably be separate tickets. If you agree can you open them and add the numbers to this ticket?

[drsm79]

metson: in fact getFTServer could use a database in couch... maybe over complicated.

[drsm79]

metson: oh, and one last thing:

1. using subprocess.PIPE for stdin, stderr and stdout can lead to deadlocks when the command is verbose ([http://docs.python.org/library/subprocess.html#convenience-functions for more info]). ftscp is probably noisey enough to be a concern, so we should look at that (e.g. write to a log that is then parsed, rather than reading the stdout)

[drsm79]

metson: (In 8920) Change main class name, add defaultProxy parameter to config, implement ftscp machinery, references #311

From: Simon simonmetson@googlemail.com

[DMWMBot]

riahi: I answer to your questions:

1- Yes, the TransferManager should be removed.
2- The defaultProxy is a valid user certificate in the server. I set this default value to be able to do tests and run transfers since a valid user certificate is needed to transfer files using ftscp as well as to remove files.
3- Yes maybe lcg-rm is better in some case. I'll open a new ticket to issue the appropriate delete command.
4- Yes, here we can use also a configuration file to have a mapping of FTS servers or get needed info from couch
5- I agree that orderFileAlgo should be replaced by changes to the CouchDB views to avoid complicated algorithms. I'll open a new ticket for this.
6- We should maybe evaluate this in an another ticket.

Cheers
Hassen

[drsm79]

metson: Ok, the other tickets are:
3: #337
4: #339
5: #338

Closing this one now.