New XrdFileCache option to invoke script on new file attach

[alja]

Add 'newfilescript' option to invoke a command when new file is opened from proxy file cache server.

This was requested by Doug Benjamin.

[abh3]

Hi Alja,

Can you give me the exact syntax of the directive and what parameters get
passed to the script?

Andy

On Tue, 2 Dec 2014, Alja Mrak-Tadel wrote:

Add 'newfilescript' option to invoke a command when new file is opened from proxy file cache server.

This was requested by Doug Benjamin.
You can merge this Pull Request by running:

git pull https://github.com/alja/xrootd doug

Or you can view, comment on it, or merge it online at:

#180

-- Commit Summary --

• Add option to execute script on new file open.
• Merge branch 'master' of github.com:xrootd/xrootd into ben
• Use XrdCl:Log instead of plain print.

-- File Changes --

M src/XrdFileCache/XrdFileCache.cc (45)
M src/XrdFileCache/XrdFileCache.hh (6)
M src/XrdFileCache/XrdFileCacheFactory.cc (31)
M src/XrdFileCache/XrdFileCacheFactory.hh (13)

-- Patch Links --

https://github.com/xrootd/xrootd/pull/180.patch
https://github.com/xrootd/xrootd/pull/180.diff

---

Reply to this email directly or view it on GitHub:
#180

[ljanyst]

Hi Alja,

this pull request contains merges. Can you please make a clean one?

Cheers,
Lukasz

[alja]

@ljanyst
I merged changes from xrootd master instead I should rebase my branch to xrootd master.

I made a new branch, cherry-picked my changes and override alja doug branch to avoid PR duplication.

[alja]

@abh3
Here is a brief descritption:

pfc.newfilescripy < path >
Invokes an executable located at given path when file is opened in proxy server. The
The executable gets two arguments: URL and cksum value.

[xrootd-dev]

Hi Alja,

You may want to cosider what happens when an rm occurs,

Andy

On Thu, 4 Dec 2014, Alja Mrak-Tadel wrote:

@abh3
Here is a brief descritption:

pfc.newfilescripy < path >
Invokes an executable located at given path when file is opened in proxy server. The
The executable gets two arguments: URL and cksum value.

---

Reply to this email directly or view it on GitHub:
#180 (comment)

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1

[alja]

@abh3
Which rm did you have in mind? Deleting the executable or remote file ?

[abh3]

Hi Alja,

Sorry for ot being precise. I meant file deletion.

Andy

On Thu, 4 Dec 2014, Alja Mrak-Tadel wrote:

@abh3
Which rm did you have in mind? Deleting the executable or remote file ?

---

Reply to this email directly or view it on GitHub:
#180 (comment)

[ljanyst]

Good, thank you, Alja! :)

[alja]

@abh3
The command is invoked when file is opened on proxy server. Saying this I think it is better the option would be called 'pfc.opencmd '.
As I understood, remove operations are forwarded to proxy origin and file is not opened in pss ofs layer.

[abh3]

Hi Alja,

Well, no. When a file is removed it goes to the pss layer. That layer
should also invoke the cache remove method. If it does not then we need to
make sure it does.

Andy

On Fri, 5 Dec 2014, Alja Mrak-Tadel wrote:

@abh3
The command is invoked when file is opened on proxy server. Saying this I think it is better the option would be called 'pfc.opencmd '.
As I understood, remove operations are forwarded to proxy origin and file is not opened in pss ofs layer.

---

Reply to this email directly or view it on GitHub:
#180 (comment)

[alja]

@abh3

Currently XrdPssSys::Unlink() only creates XrdCl:FileSystem remove request.

I don't think it is possible to propagate remove operation to XrdOucCacheIO, because this is related to a file descriptor in XrsPosix layer.
I can add virtual XrdOucCache::Unlink() function and invoke it in XrdPosixXrootd::Unlink() on XrdPosixXrootd::myCache object.

[alja]

@abh3
I guess I misunderstood Doug's request. I thought he wants to have a given command invoked only on file open event.

Is there other events too? Like Mkdir, Remdir, and Rename.

[abh3]

Hi Alja,

Indeed, if a caching layer was loaded then the remove should invoke the
caching remove prior to doing the actual remove. So, please make the
appropriate changes.

Andy

On Mon, 8 Dec 2014, Alja Mrak-Tadel wrote:

@abh3

Currently XrdPssSys::Unlink() only creates XrdCl:FileSystem remove request.

I don't think it is possible to propagate remove operation to XrdOucCacheIO, because this is related to a file descriptor in XrsPosix layer.
I can add virtual XrdOucCache::Unlink() function and invoke it in XrdPosixXrootd::Unlink() on XrdPosixXrootd::myCache object.

---

Reply to this email directly or view it on GitHub:
#180 (comment)

[abh3]

Hi Alja,

Yes, there are other events. However, I think you should just remove all
he callouts from your caching proxy and tell Doug to use the event
notfication in the OFS layer. See,

http://xrootd.org/doc/dev4/ofs_config.htm#_Toc392685004

This is what this whole thing was meant for. I don't think you need to do
anything special in the caching layer except, perhaps, a call out for a
stage-in (i.e. open r/o with file placement which the ofs callout issn't
quite suitable).

Andy

On Mon, 8 Dec 2014, Alja Mrak-Tadel wrote:

@abh3
I guess I misunderstood Doug's request. I thought he wants to have a given command invoked only on file open event.

Is there other events too? Like Mkdir, Remdir, and Rename.

---

Reply to this email directly or view it on GitHub:
#180 (comment)

[alja]

Should I close this PR ?

[abh3]

If you think the problem is solved, of course you can :-)

[alja]

@abh3

Hi Andy,

Before closing this PR and creating a new PR related to your comment #180 (comment)

I have propagated remove and rename actions from pss to cache in commit alja@6496d9c

Should the following action be propagated to cache to:
XrdPssSys::Truncate(const char *, unsigned long long, XrdOucEnv *eP=0);
?

Alja

[abh3]

Hi Alja,

Yes, truncate() should propagate into the cache (well, only for writable caches anyway).

Andy

[alja]

The feature can be implemented using OFS notification. There is no need to make change in caching proxy.