Always send PerformanceMarker for HTTP transfers

[vokac]

Similarly to the xrootd/xrootd#1785 dCache also doesn't sent performance marker for short transfers, e.g.

> COPY /atlas/atlasdatadisk/SAM/x HTTP/1.1
> Host: se1.farm.particle.cz
> User-Agent: curl/8.0.1
> Accept: */*
> Secure-Redirection: 1
> X-No-Delegate: 1
> Credentials: none
> Authorization: Bearer token1
> TransferHeaderAuthorization: Bearer token2
> Source: https://se1.farm.particle.cz/atlas/atlasdatadisk/SAM/1M
> 
< HTTP/1.1 202 Accepted
< Date: Wed, 29 Nov 2023 15:06:12 GMT
< Server: dCache/9.2.3
< Content-Type: text/perf-marker-stream
< Transfer-Encoding: chunked
< 
success: Created

as in case of longer transfer

> COPY /atlas/atlasdatadisk/SAM/x HTTP/1.1
> Host: se1.farm.particle.cz
> User-Agent: curl/8.0.1
> Accept: */*
> Secure-Redirection: 1
> X-No-Delegate: 1
> Credentials: none
> Authorization: Bearer token3
> TransferHeaderAuthorization: Bearer token4
> Source: https://se1.farm.particle.cz/atlas/atlasdatadisk/SAM/1G
> 
< HTTP/1.1 202 Accepted
< Date: Wed, 29 Nov 2023 17:52:58 GMT
< Server: dCache/9.2.3
< Content-Type: text/perf-marker-stream
< Transfer-Encoding: chunked
< 
Perf Marker
    Timestamp: 1701280383
    State: Running
    State description: Mover created
    Stripe Index: 0
    Stripe Start Time: 1701280378
    Stripe Last Transferred: 1701280383
    Stripe Transfer Time: 4
    Stripe Bytes Transferred: 391110656
    Stripe Status: RUNNING
    Total Stripe Count: 1
    RemoteConnections: tcp:[2001:718:401:6017:2::21]:20457
End
success: Created

FTS use these details to flag transfer as IPv4 vs. IPv6 vs. unknown. We are a bit stuck with #7058 which could provide even more details, because there is still not clear agreement between storage providers xrootd/xrootd#1963, but even support for this HTTP-TPC source and destination address also needs at least one Perf Marker to be produced during each file transfe.

I think we should start with simple existing case and provide at least Perf Marker with RemoteConnections for every HTTP-TPC transfer.

[paulmillar]

Please note my second point in this comment. This approach seems broken to me. The proposal in this issue (to always sending a Perf Marker) is really just a band-aid solution; something that only partially addresses a deeper problem. The problem will likely bite again!

To avoid race conditions, a short HTTP-TPC transfer would need to send connection information as part of the transfer life-cycle notification messages. These are a sequence of messages sent by the pool to indicate state changes in the transfer. The only one that could possibly work is the transfer finished message. This is the message that triggers the final success: Created or failure: REASON response line seen by the HTTP-TPC client (e.g., FTS).

If the dCache-internal transfer-finished message included transfer progress information (including connections) then the door could be updated to reply with a final perf. marker message immediately before sending the 'success/failure` response line.

Unfortunately, when the transfer completes, the dCache-internal transfer-finished message doesn't include the information needed to build a perf. marker.

Moreover, it would be hard to update dCache so the pool provides this information. In essence, the pool (undertaking the HTTP-TPC transfer) sends a transfer-complete message once the transfer has finished. At this point, there is no ongoing HTTP request (no GET and no PUT) taking place, so there's no connection(s) to report. (Similar problems exist with other information, but they are likely easier to work-around.)

The only meaningful way to solve this problem (that I can think of) would be for the pool to log all attempts to execute the HTTP request (for some specific HTTP-TPC request) and report the network connection for last attempt as part of the information included in the transfer-complete message.

This is all just to say that providing this information is non-trivial.