BBCPARC fails when the file to archive is on a remote host #19
Comments
|
Calling it a bug as this is actually supposed to work, but it doesn't currently. The only scenario where BBCP-driven archiving works is in localhost<-->localhost deployments, which is what we test with in our unit tests, and hence they have always passed. |
This is the core change needed by #19. So far we had always specified the source file with a simple file path, but to fetch remote files we need to specify them in the form [user@]host:/path/to/file. The host part is calculated with the remote IP address of the HTTP request coming from the client. This works under the assumption a connection in the reverse order can be established. Regarding the last point, bbcp seems to have options to revert the connection flow, so the source (i.e., the NGAS client) connects to the sink (i.e., the NGAS server). This *should* work in principle, but in a simple test using docker containers I had trouble making it work, and since I haven't invested more time figure this out I refrained from adding this connection flow inversion. Signed-off-by: Rodrigo Tobar <rtobar@icrar.org>
|
@gsleap I've finished pushing a first version of a fix for this under the |
|
Hi Rod, thanks for this. I've had a go and definitely have gotten further, however there is an issue with the crc checksum. When my (target) NGAS server has CRCVariant=1 in the ArchiveHandling section, then when I try to bbcparc, I get: I do have crc32c install in my NGAS virtual environment: So, I decided to ignore that and set my CRCVariant to just crc32 (CRCVariant=0) but then I get an error about CRC32z: And I get the same error about crc32z when using CRCVariant=2 as well. So, I ran the bbcp command directly on the host and can confirm that c32c and c32z do not work with bbcp. Also I think that the NGAS server is choosing c32z when CRCVariant=0 OR CRCVariant=2 but the docs say 0 should be c32 and 2 should be c32z. Manually, I can get bbcp to work with c32. |
|
Thanks for the tests @gsleap! Would you mind checking which version of bbcp do you have installed? I think you'll need at least 17.01, which contains a few fixes I contributed back regarding incorrect checksums reported by bbcp, and adding support for crc32c, so it looks like that's the issue. Apparently there are some binaries uploaded in https://www.slac.stanford.edu/~abh/bbcp/bin/, but those seem pretty old. The best is that you compile your own binary and put it in your |
|
Hi Rod, Thanks for the correct link- I had the link to v15 still! Now that I have that working I'm hitting another issue, but this seems to be a bbcp issue in what it does to check filesystem space free: Basically I have my data volume, which definitely has sufficient space in /volume1, however in the NGAS config, it is specified using the symlink I created in /home/mwa/NGAS/volume1 -> /volume1. I think bbcp is checking the /home filesystem and it defintely does not have enough free space. So instead of changing bbcp's behaviour, I stopped NGAS and edited my config file so it referred to the absolute path - i.e. /volume1. Like so: Here is an example of a stream I'm using (NOTE: it refers to the StorageSetID of "volume1" which did not changed. I just changed the MainDiskSlotId) But now, when I retry the bbcp I get an error: It's like it is looking for "/volume1" instead of "volume1" - i.e. it's looking up via the MainDiskSlotID instead of the StorageSetId. Or am I missing something? |
|
@gsleap I guess you are using a test environment? If so, would you be comfortable modifying the underlying database if required? In particular I would have a look at the contents of the Regarding bbcp, you are probably right, it's just not following symbolic links (which it should, or at least should have an option to). I'll have a look and see how difficult/easy is that to implement, and contact the authors. I've done it in the past with good results, so hopefully it won't be too much of an issue this time. |
|
Yeah totally test environment- in my db, the ngas_disks rows for that NGAS host all have mounted = 0, host_id = null, slot_id = null. The log shows the following on startup: So it looks to me like NGAS is seeing the StorageSetId in the Streams section and is looking up "MainDiskSlotId" in the StorageSets section, instead of looking for the same StorageSetId. One other work around would be to pass the "-F" parameter to bbcp to get it to ignore the free space check altogether? |
|
Thanks for the logs, that's really useful. If you can try adding the "-F" flag to the bbcp command locally (under ngas/src/ngamsServer/commands/bbcparc.py:87) that'd be awesome, but in general it'd probably be better to have bbcp do the check -- NGAS does it when not using bbcp. I'll try to figure out what's going on with the volume configuration, but that's allegedly a different problem. If the |
|
Another idea to try to workaround the volume-related error: try setting the |
|
I think that's it regarding volume directories (i.e., "slot IDs"): they apparently only work correctly if they are relative to the My recommendation would then be to try the workaround I suggested above. This is obviously a bit brutal, but should do -- otherwise mount your filesystems somewhere "safer" under Another thing: I reproduced the |
|
OK, I found the bug in bbcp, and I think I fixed it. Could you give this a try? https://github.com/ICRAR/bbcp |
|
Hi Rod, Thanks for all of your help on this and I agree- I think we are running up against issues which are maybe not exactly to do with this initial issue. I put NGAS back together as standard using symlinks and opted to try the updated ICRAR bbcp- this time I get a new error- possibly related to your new change: I also note that bbcp, because of the -z, parameter is annoyingly ignoring my IP addresses I've specified (which are 10G interfaces) and is instead doing a DNS lookup which resolves to a 1G interface instead (from the NGAS response: "redirect connection to mwax04.mwa128t.org 10.128.9.4"). Passing "-n" to bbcp would fix this, but I'm not sure how prescriptive you want NGAS to be when it calls bbcp? Maybe there should be an extra parameter in the BBCPARC command to pass optional flags to bbcp so the client can decide how they want bbcp to handle situations like this? |
|
Argh, yes, the bbcp fix wasn't perfect, it worked for me because of the way I was testing it, but now I get the same error. Just fixed it (hopefully), would you mind pulling the latest bbcp version and trying again? Yes, passing Regarding extra bbcp options, I had also thought about making these a bit more flexible. Some of the options should be set by the server though, while a few others can be given to clients, so we'll have to see how to mix both. In any case this will probably acquire less priority once all these problems are sorted. And thanks for your patience! |
|
I actually just pushed the changes I think should be required to have the bbcp transfer use the same network path, as per what I mentioned above. Could you try them out? This is still in the |
|
Hi Rod,
No worries- this is not urgent so definitely don't worry too much about it if you get busy with other things.
I've pulled the newest ICRAR/bbcp and this now works, but as we suspected, it ignored my hardcoded IP addresses provided in the BBCPARC command and instead transferred via the 1G interface which corresponds to the hostname.
Here is my Curl command which I execute on the server which has the source file (192.168.120.204 / mwax04.mwa128t.org) I want to archive to the destination server (192.168.120.110 / mwacache10.mwa128t.org):
curl "http://192.168.120.110:7777/BBCPARC?filename=mwa@192.168.120.204:/data/20191210/rawdump_1260043216.raw&bnum_streams=12&mime_type=application%2Fx-mwa-fits"
Now here is exactly what NGAS executes on the destination NGAS sever:
2020-01-28T06:52:11.783 [16979] [ R-2] [ INFO] ngamsServer.commands.bbcparc#bbcpFile:90 Executing external command: bbcp -f -V -S "ssh -x -a -oBatchMode=yes -oGSSAPIAuthentication=no -oFallBackToRsh=no %4 %I -l %U %H bbcp" -e -E c32c=/dev/stdout -s 12 -P 2 -z mwa@192.168.120.204:/data/20191210/rawdump_1260043216.raw /home/mwa/NGAS/volume4/staging/NGAMS_TMP_FILE___d7v7_jbdrawdump_1260043216.raw.fits
So, I thought the reason it is using the 1G link is the "%H" directive used by NGAS in the remote ssh settings to execute bbcp remotely. %H substitutes in the hostname, but in this case we don't want that at all since it resolves to the 1G IP, we want to use whatever IP the user gave us. However hacking this change into the NGAS source, rebuilding it and restarting NGAS results in the same 1G interface being used. I think something in BBCP is still doing a hostname lookup and using it.
So I've taken your advice and removed -z, rebuilt and retried and now I get this error:
<?xml version="1.0" ?>
<!DOCTYPE NgamsStatus SYSTEM "http://mwacache10.mwa128t.org:7777/RETRIEVE?internal=ngamsStatus.dtd">
<NgamsStatus>
<Status Date="2020-01-28T08:05:22.230" HostId="mwacache10:7777" Message="bbcp returncode: 3. Command line: ['bbcp -f -V -S "ssh -x -a -oBatchMode=yes -oGSSAPIAuthentication=no -oFallBackToRsh=no %4 %I -l %U -%H bbcp" -e -E c32c=/dev/stdout -s 12 -P 2 "" mwa@192.168.120.204:/data/20191210/rawdump_1260043216.raw /home/mwa/NGAS/volume2/staging/NGAMS_TMP_FILE___am7oimv6rawdump_1260043216.raw.fits'], out: b'', err: b'bbcp: Host 127.0.1.1 redirect connection to mwacache10.mwa128t.org 127.0.1.1\nbbcp: Source file not specified.\nbbcp: Accept timed out on port 5031\nbbcp: Unable to allocate more than 0 of 12 data streams.\n'" State="ONLINE" Status="FAILURE" SubState="IDLE" Version="11.0/2018-10-26T07:00:00"/>
So this also makes it look like bbcp is doing a lookup when really I don't think it should at all. Maybe I'll stick to QARCHIVE for the time being ;-)
Cheers
Greg
…________________________________
From: rtobar <notifications@github.com>
Sent: Friday, 24 January 2020 3:23 PM
To: ICRAR/ngas <ngas@noreply.github.com>
Cc: Greg Sleap <greg.sleap@curtin.edu.au>; Mention <mention@noreply.github.com>
Subject: Re: [ICRAR/ngas] BBCPARC fails when the file to archive is on a remote host (#19)
I actually just pushed the changes I think should be required to have the bbcp transfer use the same network path, as per what I mentioned above. Could you try them out? This is still in the bbcp_fixes branch.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#19?email_source=notifications&email_token=AE2L5FQZ7TQ6QELVWH7RFZ3Q7KJNRA5CNFSM4JWLSLBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJZ6K7A#issuecomment-578020732>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AE2L5FU7SIUYHBEZNZOBNU3Q7KJNRANCNFSM4JWLSLBA>.
|
|
Hi @gsleap! Thanks for the update. Did you try also pulling from the latest |
|
Ooops, sorry about that- I missed that!
Ok, I pulled the latest bbcp_fixes, rebuilt, restarted and retested and got this result:
(mwax)mwa@mwax04:~$ curl "http://192.168.120.110:7777/BBCPARC?filename=mwa@192.168.120.204:/data/20191210/rawdump_1260043216.raw&bnum_streams=12&mime_type=application%2Fx-mwa-fits"
<?xml version="1.0" ?>
<!DOCTYPE NgamsStatus SYSTEM "http://mwacache10.mwa128t.org:7777/RETRIEVE?internal=ngamsStatus.dtd">
<NgamsStatus>
<Status Date="2020-01-28T08:25:07.458" HostId="mwacache10:7777" Message="bbcp returncode: 255. Command line: ['bbcp -f -V -z -S "ssh -x -a -oBatchMode=yes -oGSSAPIAuthentication=no -oFallBackToRsh=no %4 %I -l %U %H bbcp" -e -E c32c=/dev/stdout -s 12 -P 2 -z mwa@192.168.120.204:/data/20191210/rawdump_1260043216.raw 192.168.120.110:7777:/home/mwa/NGAS/volume2/staging/NGAMS_TMP_FILE___3airu2emrawdump_1260043216.raw.fits'], out: b'', err: b'bbcp: Host [::ffff:192.168.120.204] redirect connection to mwax04.mwa128t.org 10.128.9.4\nHost key verification failed.\r\n'" State="ONLINE" Status="FAILURE" SubState="IDLE" Version="11.0/2018-10-26T07:00:00"/>
There seems to be some ipv6 in there now "Host [::ffff:192.168.120.204]" as well as "redirect connection to mwax04.mwa128t.org 10.128.9.4\nHost key verification failed."
Cheers
Greg
…________________________________
From: rtobar <notifications@github.com>
Sent: Tuesday, 28 January 2020 4:17 PM
To: ICRAR/ngas <ngas@noreply.github.com>
Cc: Greg Sleap <greg.sleap@curtin.edu.au>; Mention <mention@noreply.github.com>
Subject: Re: [ICRAR/ngas] BBCPARC fails when the file to archive is on a remote host (#19)
Hi @gsleap<https://github.com/gsleap>! Thanks for the update. Did you try also pulling from the latest bbcp_fixes in the NGAS repo? I put a change there last week that might finally resolve the way the bbcp source is trying to contact the sink (i.e., it should use the same IP that the NGAS client used to contact the NGAS server). The giveaway is the bbcp command line: the final argument shouldn't be just a filename, but something like /home/mwa/NGAS/volume2/staging/NGAMS_TMP_FILE___am7oimv6rawdump_1260043216.raw.fits but like 192.168.120.110:/home/mwa/NGAS/volume2/staging/NGAMS_TMP_FILE___am7oimv6rawdump_1260043216.raw.fits, which I was hoping would make the bbcp source connect through the correct network interface.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#19?email_source=notifications&email_token=AE2L5FWMSWORFCUUJFWJ7QLQ77S3LA5CNFSM4JWLSLBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKCNKDQ#issuecomment-579130638>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AE2L5FSZXE6TX5FD2ZKM67LQ77S3LANCNFSM4JWLSLBA>.
|
|
Mmmm, would you mind as a final test to try this out on the NGAS server machine?
The only difference with the latest bbcp command generated by NGAS is that the target filename doesn't have the |
|
Hi Rod,
Sorry for the delay:
Ok, first try:
bbcp: Permission denied opening checksum file /dev/stdout
So I removed the "=/dev/stdout" from the -E c32c parameter:
The authenticity of host '192.168.120.110 (192.168.120.110)' can't be established.
ECDSA key fingerprint is SHA256:JL+b82wMq4P+FLrVPNIxiYaz3yRaey2EOk3HEimn1jo.
Are you sure you want to continue connecting (yes/no)?
So I typed in yes, but then I got:
mwa@192.168.120.110's password:
hmm, so this is looking like we are trying to connect in to ourself! So, it seemed weird, but I just added by public key to .ssh/authorized_keys and then it did not have any errors, but it still is using the 1G interface 10.128.9.4:
(ngas_rt) mwa@mwacache10:~$ bbcp -f -V -z -S "ssh -x -a -oBatchMode=yes -oGSSAPIAuthentication=no -oFallBackToRsh=no %4 %I -l %U %H bbcp" -e -E c32c -s 12 -P 2 -z mwa@192.168.120.204:/data/20191210/rawdump_1260043216.raw 192.168.120.110:/home/mwa/NGAS/volume2/staging/NGAMS_TMP_FILE___3airu2emrawdump_1260043216.raw.fits
bbcp: Host [::ffff:192.168.120.204] redirect connection to mwax04.mwa128t.org 10.128.9.4
Target mwacache10.mwa128t.org using initial recv window of 374400
Source mwax04.mwa128t.org using initial send window of 87380
bbcp: Creating /home/mwa/NGAS/volume2/staging/NGAMS_TMP_FILE___3airu2emrawdump_1260043216.raw.fits
bbcp: 200130 10:20:08 0% done; 112.9 MB/s, avg 112.9 MB/s
bbcp: 200130 10:20:09 0% done; 113.0 MB/s, avg 112.9 MB/s
bbcp: 200130 10:20:10 0% done; 113.2 MB/s, avg 113.0 MB/s
...
^C
Something is still trying to redirect via the 1G link it seems?
Cheers
Greg
…________________________________
From: rtobar <notifications@github.com>
Sent: Tuesday, 28 January 2020 4:47 PM
To: ICRAR/ngas <ngas@noreply.github.com>
Cc: Greg Sleap <greg.sleap@curtin.edu.au>; Mention <mention@noreply.github.com>
Subject: Re: [ICRAR/ngas] BBCPARC fails when the file to archive is on a remote host (#19)
Mmmm, would you mind as a final test to try this out on the NGAS server machine?
bbcp -f -V -z -S "ssh -x -a -oBatchMode=yes -oGSSAPIAuthentication=no -oFallBackToRsh=no %4 %I -l %U %H bbcp" -e -E c32c=/dev/stdout -s 12 -P 2 -z mwa@192.168.120.204:/data/20191210/rawdump_1260043216.raw 192.168.120.110:/home/mwa/NGAS/volume2/staging/NGAMS_TMP_FILE___3airu2emrawdump_1260043216.raw.fits
The only difference with the latest bbcp command generated by NGAS is that the target filename doesn't have the :7777 (I didn't think of that). If that works then we would be quote close to the "final" result; otherwise I'll have to do some more experimentation on my side.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#19?email_source=notifications&email_token=AE2L5FXNNIYET7DBNR2FFATQ77WKNA5CNFSM4JWLSLBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKCPWDI#issuecomment-579140365>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AE2L5FSW5DKWFRYWFONTVP3Q77WKNANCNFSM4JWLSLBA>.
|
|
@gsleap yes, it seems so. Sorry for all this trial and error, and all the effort you've been putting into this, but after reading more of the bbcp code, this seems to be a limitation of bbcp: the hostname on each node is the main piece of information used by the source to establish the data exchange connections between the bbcp instances. However, if I'm reading this correctly, it might be possible (but I'm not fully sure) to bypass this behavior if you are using the So once again I must ask ask more from you. Would you mind trying the last command line, but with the |
|
Hi Rod,
I was going to apologise to *you* for all this back and forth! 😉 Anyway it's no problem, it looks like although simple on the surface, using bbcp is much trickier than anyone realised!
Here is my output:
bbcp: Connection refused unable to connect to port mwacache10.mwa128t.org:5031
bbcp: Unable to allocate more than 0 of 12 data streams.
So I *think* what's happening is:
* bbcp on mwacache10 is sshing to mwax04 to launch bbcp and it is run to listen on the 10G interface
* Then bbcp on mwax04 is then trying to communicate to bbcp on mwacache10, but via a name lookup of the host which results in trying to connect on the 1G interface which results in connection refused (because it isn't listening on that interface).
I think that's right- it gets confusing with -z and knowing which is the server and which is the client!
Cheers
Greg
…________________________________
From: rtobar <notifications@github.com>
Sent: Thursday, 30 January 2020 3:03 PM
To: ICRAR/ngas <ngas@noreply.github.com>
Cc: Greg Sleap <greg.sleap@curtin.edu.au>; Mention <mention@noreply.github.com>
Subject: Re: [ICRAR/ngas] BBCPARC fails when the file to archive is on a remote host (#19)
@gsleap<https://github.com/gsleap> yes, it seems so. Sorry for all this trial and error, and all the effort you've been putting into this, but after reading more of the bbcp code, this seems to be a limitation of bbcp: the hostname on each node is the main piece of information used by the source to establish the data exchange connections between the bbcp instances. However, if I'm reading this correctly, it might be possible (but I'm not fully sure) to bypass this behavior if you are using the -n (no DNS) option, which sides a few side effects, this included.
So once again I must ask ask more from you. Would you mind trying the last command line, but with the -n option?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#19?email_source=notifications&email_token=AE2L5FXJXFQGFYP4ES45AJLRAJ3VNA5CNFSM4JWLSLBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKJ46EA#issuecomment-580112144>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AE2L5FXHNFZTTH3ALPD2EJ3RAJ3VNANCNFSM4JWLSLBA>.
|
Indeed, who would have guessed! OK, another try, this time without
I think the |
|
Hi Rod,
haha, yes indeed!
Ok, giving that a go (once again from the ngas/target side):
(ngas_rt) mwa@mwacache10:~$ bbcp -f -V -n -S "ssh -x -a -oBatchMode=yes -oGSSAPIAuthentication=no -oFallBackToRsh=no %4 %I -l %U %H bbcp" -e -E c32c -s 12 -P 2 mwa@192.168.120.204:/data/20191210/rawdump_1260043216.raw 192.168.120.110:/home/mwa/NGAS/volume2/staging/NGAMS_TMP_FILE___3airu2emrawdump_1260043216.raw.fits
bbcp: Connection refused unable to connect to port mwax04.mwa128t.org:5031
bbcp: Unable to allocate more than 0 of 12 data streams.
bbcp: Accept timed out on port 5031
bbcp: Unable to allocate more than 0 of 12 data streams.
Once again, it keeps wanting to use the fqdn not the IP we gave it and there is nothing listening on that interface (and in any case we don't want to xfer on the 1G anyway when we have a big fat 10G pipe available).
Cheers
Greg
…________________________________
From: rtobar <notifications@github.com>
Sent: Friday, 31 January 2020 3:12 PM
To: ICRAR/ngas <ngas@noreply.github.com>
Cc: Greg Sleap <greg.sleap@curtin.edu.au>; Mention <mention@noreply.github.com>
Subject: Re: [ICRAR/ngas] BBCPARC fails when the file to archive is on a remote host (#19)
it looks like although simple on the surface, using bbcp is much trickier than anyone realised!
Indeed, who would have guessed!
OK, another try, this time without -z, but with -n:
bbcp -f -V -n -S "ssh -x -a -oBatchMode=yes -oGSSAPIAuthentication=no -oFallBackToRsh=no %4 %I -l %U %H bbcp" -e -E c32c -s 12 -P 2 mwa@192.168.120.204:/data/20191210/rawdump_1260043216.raw 192.168.120.110:/home/mwa/NGAS/volume2/staging/NGAMS_TMP_FILE___3airu2emrawdump_1260043216.raw.fits
I think the -z shouldn't have been there in the first place, my bad; it causes the bbcp SINK (mwacache10) to connect to the bbcp SOURCE (mwax04), but we definitely want the opposite -- and using the same IP that the NGAS client in mwax04 used to connect to NGAS in mwacache10. Fingers crossed...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#19?email_source=notifications&email_token=AE2L5FUV6B7F2ZRU4GWDCALRAPFNNA5CNFSM4JWLSLBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKNXDSY#issuecomment-580612555>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AE2L5FV4GFWKJVUDR7JWJLLRAPFNNANCNFSM4JWLSLBA>.
|
|
Tes, bbcp is still basing its connectivity on name resolutions, and doesn't seem to allow for specific interfaces to be used. Too bad :(. The next step would be to delve a bit deeper into bbcp and try to add such support. That's a bit outside of the scope of this issue though, which has been satisfactorily resolved I'd say? If you're OK with it, I'll be closing this issue as the original problem is long gone, and will open a new one to keep investigate at some point the bbcp implementation and see how easy/hard would it be to add the missing functionality. |
|
Agreed, this is different issue- the one involving ngas was that it was not allowing transfers outside of the localhost test scenario. This is now fixed. Cheers. |
|
Created #21 for keeping track of the network path issues with bbcp, and closing this one now. |
This is the core change needed by #19. So far we had always specified the source file with a simple file path, but to fetch remote files we need to specify them in the form [user@]host:/path/to/file. The host part is calculated with the remote IP address of the HTTP request coming from the client. This works under the assumption a connection in the reverse order can be established. Regarding the last point, bbcp seems to have options to revert the connection flow, so the source (i.e., the NGAS client) connects to the sink (i.e., the NGAS server). This *should* work in principle, but in a simple test using docker containers I had trouble making it work, and since I haven't invested more time figure this out I refrained from adding this connection flow inversion. Signed-off-by: Rodrigo Tobar <rtobar@icrar.org>
|
Just merged this code into the |
When issuing a BBCPARC command:
http://ngashost:7777/BBCPARC?filename=user%40hostwithfile%3A%2Fdata%2F1247842824_20190722150006_ch114_000.fits&bnum_streams=12&mime_type=application%2Fx-mwa-fits
NGAS server returns 400 BAD REQUEST, which error as shown in log file as below:
2019-12-06T05:01:03.292 [ 5303] [ R-65] [ INFO] ngamsServer.ngamsServer#handleHttpRequest:1696 Handling HTTP request: client_address=('192.168.120.204', 43008) - method=GET - path=|BBCPARC?filename=user%40hostwithfile%3A%2Fdata%2F1247842824_20190722150006_ch114_000.fits&bnum_streams=12&mime_type=application%2Fx-mwa-fits|
2019-12-06T05:01:03.293 [ 5303] [ R-65] [ INFO] ngamsServer.ngamsCmdHandling#_get_module:74 Received command: BBCPARC
2019-12-06T05:01:03.293 [ 5303] [ R-65] [ INFO] ngamsServer.ngamsArchiveUtils#_dataHandler:1055 Handling archive pull request
2019-12-06T05:01:03.294 [ 5303] [ R-65] [ ERROR] ngamsServer.ngamsServer#reqCallBack:1633 Error while serving request
Traceback (most recent call last):
File "/usr/lib/python3.6/urllib/request.py", line 1474, in open_local_file
stats = os.stat(localfile)
FileNotFoundError: [Errno 2] No such file or directory: '/user@hostwithfile:/data/1247842824_20190722150006_ch114_000.fits'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mwa/ngas_rt/lib/python3.6/site-packages/ngamsServer-11.0-py3.6.egg/ngamsServer/ngamsServer.py", line 1617, in reqCallBack
method, path, headers)
File "/home/mwa/ngas_rt/lib/python3.6/site-packages/ngamsServer-11.0-py3.6.egg/ngamsServer/ngamsServer.py", line 1711, in handleHttpRequest
ngamsCmdHandling.handle_cmd(self, reqPropsObj, httpRef)
File "/home/mwa/ngas_rt/lib/python3.6/site-packages/ngamsServer-11.0-py3.6.egg/ngamsServer/ngamsCmdHandling.py", line 63, in handle_cmd
msg = _get_module(srvObj, reqPropsObj).handleCmd(srvObj, reqPropsObj, httpRef)
File "/home/mwa/ngas_rt/lib/python3.6/site-packages/ngamsServer-11.0-py3.6.egg/ngamsServer/commands/bbcparc.py", line 182, in handleCmd
transfer=bbcp_transfer)
File "/home/mwa/ngas_rt/lib/python3.6/site-packages/ngamsServer-11.0-py3.6.egg/ngamsServer/ngamsArchiveUtils.py", line 1033, in dataHandler
do_replication=do_replication, transfer=transfer)
File "/home/mwa/ngas_rt/lib/python3.6/site-packages/ngamsServer-11.0-py3.6.egg/ngamsServer/ngamsArchiveUtils.py", line 1062, in _dataHandler
handle = urlrequest.urlopen(url)
File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/usr/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib/python3.6/urllib/request.py", line 1452, in file_open
return self.open_local_file(req)
File "/usr/lib/python3.6/urllib/request.py", line 1491, in open_local_file
raise URLError(exp)
urllib.error.URLError: <urlopen error [Errno 2] No such file or directory: '/user@hostwithfile:/data/1247842824_20190722150006_ch114_000.fits'>
NOTE: this is not a showstopper, as I can easily use QARCHIVE, but I thought I would try out bbcparc first.
The text was updated successfully, but these errors were encountered: