Skip to content

Commit

Permalink
Calculate remote file to transfer via bbcp
Browse files Browse the repository at this point in the history
This is the core change needed by #19. So far we had always specified
the source file with a simple file path, but to fetch remote files we
need to specify them in the form [user@]host:/path/to/file. The host
part is calculated with the remote IP address of the HTTP request coming
from the client. This works under the assumption a connection in the
reverse order can be established.

Regarding the last point, bbcp seems to have options to revert the
connection flow, so the source (i.e., the NGAS client) connects to the
sink (i.e., the NGAS server). This *should* work in principle, but in a
simple test using docker containers I had trouble making it work, and
since I haven't invested more time figure this out I refrained from
adding this connection flow inversion.

Signed-off-by: Rodrigo Tobar <rtobar@icrar.org>
  • Loading branch information
rtobar committed Feb 3, 2020
1 parent 8bba0db commit 5e79964
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 1 deletion.
18 changes: 17 additions & 1 deletion src/ngamsServer/ngamsServer/commands/bbcparc.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,13 +144,29 @@ def get_params(request):
return bbcp_param(port, winsize, num_streams, checksum)


def get_source_file(request):
"""
Put the NGAS client address as the netloc of the URL in the source file,
so that bbcp contacts the host issuing the HTTP request. Carry over any
username, if given
"""
url = urlparse.urlsplit(request.getFileUri())
if not url.scheme:
url = urlparse.urlsplit('file://' + request.getFileUri())
if url.username:
netloc = url.username + '@' + request.client_addr
else:
netloc = request.client_addr
return netloc + ':' + url.path

def bbcp_transfer(request, out_fname, crc_name, skip_crc):

bparam = get_params(request)
source_file = get_source_file(request)

# perform the bbcp transfer, we will always return the checksum
start = time.time()
checksum = bbcpFile(request.getFileUri(), out_fname, bparam, crc_name, skip_crc)
checksum = bbcpFile(source_file, out_fname, bparam, crc_name, skip_crc)
size = getFileSize(out_fname)
totaltime = time.time() - start

Expand Down
1 change: 1 addition & 0 deletions src/ngamsServer/ngamsServer/ngamsServer.py
Original file line number Diff line number Diff line change
Expand Up @@ -1712,6 +1712,7 @@ def handleHttpRequest(self,
msg = "Handling HTTP request: client_address=%s - method=%s - path=|%s|"
logger.info(msg, str(clientAddress), method, safePath)

reqPropsObj.client_addr = clientAddress[0]
reqPropsObj.unpackHttpInfo(self.getCfg(), method, path, headers)

try:
Expand Down

0 comments on commit 5e79964

Please sign in to comment.