[XrdHttp and XrdHttpTPC] Pass packet marking information from HTTP headers to the XRootD layer

[ccaffy]

XRootD supports packet marking with UDP firefly: https://xrootd.slac.stanford.edu/doc/dev56/xrd_config.htm#_Toc135684345

This functionality works with XrootD. The user can pass the information about the experiment and its activity by specifying in the opaque data of a transfer the following key/value: scitag.flow=<experimentID><<6|<activityID> .

For HTTP TPC transfer, the user will pass these information via specific headers as described here: https://docs.google.com/document/d/1x9JsZ7iTj44Ta06IHdkwpv5Q2u4U2QGLWnUeN2Zf5ts/edit#heading=h.ksbp08p3u6lb

• TransferFlowExperiment
• TransferHeaderFlowExperiment
• TransferFlowActivity
• TransferHeaderFlowActivity

I believe I just need to pass the information coming from the headers to the xroot layer by appending scitag.flow=expe|activity ?

[abh3]

Almost. I added an objection comment as the current spec is not at all close to what we agreed to earlier in the year. So, I would hold off implementing anything.

…

On Fri, 2 Jun 2023, ccaffy wrote: XRootD supports packet marking with UDP firefly: https://xrootd.slac.stanford.edu/doc/dev56/xrd_config.htm#_Toc135684345 This functionality works with XrootD. The user can pass the information about the experiment and its activity by specifying in the opaque data of a transfer the following key/value: `scitag.flow=<experimentID><<6|<activityID>` . For HTTP TPC transfer, the user will pass these information via specific headers as described here: https://docs.google.com/document/d/1x9JsZ7iTj44Ta06IHdkwpv5Q2u4U2QGLWnUeN2Zf5ts/edit#heading=h.ksbp08p3u6lb - `TransferFlowExperiment` - `TransferHeaderFlowExperiment` - `TransferFlowActivity` - `TransferHeaderFlowActivity` I believe I just need to pass the information coming from the headers to the xroot layer by appending `scitag.flow=expe|activity` ? -- Reply to this email directly or view it on GitHub: #2024 You are receiving this because you are subscribed to this thread. Message ID: ***@***.***>

[ccaffy]

Thanks, I'll back off then. Let's use this issue to track the progress of this topic.

[bbockelm]

@ccaffy - setting aside the question headers, I don't think the implementation is that simple once fireflies are requested.

For XrdHttp: the xrootd protocol handler only sees the requests to/from memory. I don't think it has enough information to create the fireflies -- the connection information is only known at the HTTP layer so the HTTP layer needs to be sending the fireflies.

For HTTP-TPC: Similar situation. The socket is only known to libcurl. You'd need the fireflies sent at the main loop handling the transfer.

So, while we're waiting on the resolution of the header values, you can probably still look at how to best construct the fireflies using the existing classes.

[abh3]

Actually, the original design would handle this. It's a matter of getting the headers promoted the right way. There is some disagreement in the working group about what the headers should be. We are in discussion now. Once we get that settled we will know what direction we need to take.

…

On Mon, 5 Jun 2023, Brian P Bockelman wrote: @ccaffy - setting aside the question headers, I don't think the implementation is that simple once fireflies are requested. For XrdHttp: the xrootd protocol handler only sees the requests to/from memory. I don't think it has enough information to create the fireflies -- the connection information is only known at the HTTP layer so the HTTP layer needs to be sending the fireflies. For HTTP-TPC: Similar situation. The socket is only known to libcurl. You'd need the fireflies sent at the main loop handling the transfer. So, while we're waiting on the resolution of the header values, you can probably still look at how to best construct the fireflies using the existing classes. -- Reply to this email directly or view it on GitHub: #2024 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[bbockelm]

Actually, the original design would handle this

I don't understand this comment. Let's ignore the headers for a second -- how are you planning on implementing the fireflies / packet marking. None of the shared layers see the HTTP socket so, while we could reuse quite a bit of logic, there's custom handling and sending of firefly packets needed, right?

[xrootd-dev]

The whole think is portably architected. So, in the case of TPC it's a matter of a call. In the case of anything else, it's just happens in the workflow.

…

On Wed, 7 Jun 2023, Brian P Bockelman wrote: > Actually, the original design would handle this I don't understand this comment. Let's ignore the headers for a second -- how are you planning on implementing the fireflies / packet marking. None of the shared layers see the HTTP socket so, while we could reuse quite a bit of logic, there's custom handling and sending of firefly packets needed, right? -- Reply to this email directly or view it on GitHub: #2024 (comment) You are receiving this because you are subscribed to this thread. Message ID: ***@***.***> ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1

[ccaffy]

Done and merged via #2109