Skip to content

Commit

Permalink
ftp: fix on-the-fly checksum calculation
Browse files Browse the repository at this point in the history
Motivation:

Globus transfer service requires a GridFTP endpoint to provide an MD5
checksum of files in order to support data integrity checking.  By
default, dCache calculates only ADLER32 checksum, so the ftp door was
updated to support dynamic checksum calculation.

Unfortunately, this failed to take into account the 'ftp.net.internal'
configuration property, which allows an admin to specify on which
interface the door should ask pools to connect when relaying data.

Worse, the current behaviour (which hard-codes a listen on all
interfaces) tells the pool to connect to the door on port 0.0.0.0 (IPv4)
or :: (IPv6), which will only work if the pool and door are running on
the same machine.

Modification:

Honour the 'ftp.net.internal' setting when dynamically calculating a
checksum.

Fix broken dynamic checksum calculation when binding the server socket
to all interfaces.

Result:

Fix Globus transfers hanging when "verify file integrity after transfer"
is enabled and the pool is not configured to calculate an MD5 checksum.

Honour the 'ftp.net.internal' configuration property for dynamic
checksum calculation.

Target: master
Request: 7.0
Request: 6.2
Request: 6.1
Request: 6.0
Request: 5.2
Requires-notes: yes
Requires-book: no
Patch: https://rb.dcache.org/r/12763/
Acked-by: Tigran Mkrtchyan
  • Loading branch information
paulmillar committed Jan 19, 2021
1 parent 7664bd9 commit 7eaf3a7
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 3 deletions.
Expand Up @@ -2941,7 +2941,9 @@ public void doCksm(String algo, String path, long offsetL, long lengthL)
.orElse(null);

if (checksum == null) {
ChecksumCalculatingTransfer cct = new ChecksumCalculatingTransfer(_pnfs, _subject, _authz, absPath, type, new PortRange(0,0));
ChecksumCalculatingTransfer cct = new ChecksumCalculatingTransfer(_pnfs,
_subject, _authz, absPath, type, _internalInetAddress,
new PortRange(0,0));
setTransfer(cct);
TimerTask sendProgress = null;
try {
Expand Down
Expand Up @@ -6,6 +6,7 @@
import javax.security.auth.Subject;

import java.io.IOException;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.ServerSocketChannel;
Expand Down Expand Up @@ -41,24 +42,26 @@ public class ChecksumCalculatingTransfer extends Transfer

private final ChecksumType desiredType;
private final PortRange portRange;
private final InetAddress localAddress;

private long calculated;

public ChecksumCalculatingTransfer(PnfsHandler pnfs, Subject subject,
Restriction namespaceRestriction, FsPath path, ChecksumType type,
PortRange range)
InetAddress localAddress, PortRange range)
{
super(pnfs, subject, namespaceRestriction, path);
desiredType = type;
portRange = range;
this.localAddress = localAddress;
}

public Checksum calculateChecksum() throws CacheException,
InterruptedException, IOException, NoSuchAlgorithmException
{
boolean success = false;
ServerSocketChannel ssc = ServerSocketChannel.open();
portRange.bind(ssc.socket());
portRange.bind(ssc.socket(), localAddress);
setAdditionalAttributes(EnumSet.of(FileAttribute.CHECKSUM));
readNameSpaceEntry(false);
LOGGER.debug("calculating checksum using port {}", ssc.getLocalAddress());
Expand Down

0 comments on commit 7eaf3a7

Please sign in to comment.