Third-party-copy plugin for XrdHttp #679
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Only a minimal proof-of-concept to show we can get COPY intercepted and working.
This provides a working implementation of the push mode for COPY. Additionally, this now compiles against the latest Xrootd build, implements overwrite mode, and correctly copies headers into the destination transfer. As of this tag, a Xrootd server can copy files to and from itself.
With this, both push and pull mode can involve redirections.
Primarily from the documentation here: https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/WebDAV/Extensions#ThirdPartyCopies and the contents of various personal email threads.
With the chunked transfer-encoding in the Xrootd external handler framework, we can return HTTP performance markers to the client.
If Xrootd supports chunked transfer encoding, then send performance markers along with the COPY request response.
libdavix only parser transfer marker information once every 1024 characters (which caused us to create overly-chatty markers). This removes that workaround as a fix is available from upstream libdavix.
Knowing the size of the remote resource is the first step in doing a multi-stream HTTP GET transfer.
Split tpc.cpp to several helper files. This is done in preparation of the addition of multi-streaming support, which will make the implementation larger and more complex.
Currently, there is a one-to-one correspondence between a HTTP request (a state object) and a stream going out to a Xrootd file handle. In preparation for multiple streams, we want to split up the concept of a request and a file stream. Multiple streams will be implemented by doing multiple large requests going to the same file.
Add skeleton of the data re-ordering infrastructure. As there's only a single-stream (no re-order necessary), none of this will get invoked for now.
- Do not use POSC as this requires some fchmod support we don't have. - Log the URL utilized for pull requests.
|
If it's not clear, this is NOT meant for 4.8.2. It might be worthwhile to merge relatively soon afterward. |
|
Thank you very much Brian. We will surely use it a lot |
simonmichal
added a commit
that referenced
this pull request
May 8, 2018
This reverts commit e3550ce, reversing changes made to a768fd6. It has been decided that this PR is to be reverted as it does not follow XRootD coding conventions and brakes the build on vanilla RHEL6 platform in presence of libcurl-devel package. There should be a check in the CMake infrastructures that ensures the XrdTpc package will be only built if a proper version of libcurl-devel is present (or XrdTpc should built with version of libcurl-devel provided in raw RHEL6 environment). Moreover the new component should be packaged properly (if neccessary a _with_tpchttp macro should be tested in the spec file).
|
Hi guys, The PR does not follow XRootD coding conventions and brakes the build on vanilla RHEL6 platform in presence of libcurl-devel package. There should be a check in the CMake infrastructures that ensures the XrdTpc package will be only built if a proper version of libcurl-devel is present (or XrdTpc should built with version of libcurl-devel provided in raw RHEL6 environment). Moreover the new component should be packaged properly (if necessary a _with_tpchttp macro should be tested in the spec file). @abh3 can provide additional detail. |
bbockelm
added a commit
to bbockelm/xrootd
that referenced
this pull request
Jun 7, 2018
…c-rename"" This reverts commit d46c516. Reverting-the-revert to start working on cleanup of this branch for C++03 support and eventual re-merge.
simonmichal
pushed a commit
that referenced
this pull request
Jul 13, 2018
…me"" This reverts commit d46c516. Reverting-the-revert to start working on cleanup of this branch for C++03 support and eventual re-merge.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR integrates the external xrootd-tpc plugin (see https://github.com/bbockelm/xrootd-tpc) into the main Xrootd source itself.
The plugin will be built if HTTP is enabled and, when loaded, will handle the COPY WebDAV verb via a callout to
libcurl. Multistreaming is supported, as well various extensions used by the WLCG community (Copy-Headersupport to add specific headers to the TPC).To enable, one can add the following to the configuration file: