Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make ftp connections persistent for 30 seconds #15

Merged
merged 4 commits into from Jan 21, 2019

Conversation

@mraspaud
Copy link
Member

commented Aug 10, 2018

No description provided.

mraspaud added 2 commits Aug 10, 2018
Defaults to 30 seconds for configuring the connection persistence time
@TAlonglong

This comment has been minimized.

Copy link
Collaborator

commented Aug 11, 2018

Really nice.

If I understands correctly: If a connection attempt for some reason raise an exception, this is caught in the move_it mover try - exception loop and the copy attempt is abandoned?

Trygve Aspenes

@mraspaud

This comment has been minimized.

Copy link
Member Author

commented Aug 13, 2018

yes, exactly. The error is then sent as a message so that the client is aware of the problem, and iirc, the client will go on to the next server providing the file to retrieve it.

@TAlonglong

This comment has been minimized.

Copy link
Collaborator

commented on bf6c4f5 Nov 7, 2018

What does mkdfile do? I guess it should be some ftplib command, but I cant find it.

And, why do you want to quit the connection after the copy? I thought the idea was to keep the connection?
I guess it's some details I don't read out of the code.

This comment has been minimized.

Copy link
Member Author

replied Nov 7, 2018

Good point, that's a bug (the connection closing).
As for mkdfile, I'll have to look it up.

This comment has been minimized.

Copy link
Collaborator

replied Dec 5, 2018

@mraspaud what I'm testing is my implementation of the persistent connection with ssh. So of course my implementation can have issues

On Monday this week my trollmoves system broke down again. There I saw (or I think I saw) that the connection was lost outside trollmoves, but trollmoves still saw it as a valid connection. I will try to find back the logs. So this might be an issue.

[2018-12-04 11:40:41,882 DEBUG trollmoves.server] Copying to: scp://xvis-m3b/lustre/storeA/project/metproduction/products/satdata_polar/pps-v2014/ears/S_NWC_CT_m
etopa_62920_20181204T1044000Z_20181204T1107599Z.nc
[2018-12-04 11:40:41,882 DEBUG trollmoves.server] Getting connection to remotesensing@xvis-m3b:None
[2018-12-04 11:40:41,882 DEBUG trollmoves.server] checking ssh connection ... True
[2018-12-04 11:40:41,883 DEBUG paramiko.transport] [chan 3] Max packet in: 32768 bytes
[2018-12-04 11:40:41,884 DEBUG paramiko.transport] [chan 3] Max packet out: 32768 bytes
[2018-12-04 11:40:41,885 DEBUG paramiko.transport] Secsh channel 3 opened.
[2018-12-04 11:40:41,891 DEBUG paramiko.transport] [chan 3] EOF sent (3)
[2018-12-04 11:40:41,892 ERROR trollmoves.server] Something went wrong with scp: Channel closed.
[2018-12-04 11:40:41,892 ERROR trollmoves.server] Exception name SSHException
[2018-12-04 11:40:41,892 ERROR trollmoves.server] Exception args ('Channel closed.',)
[2018-12-04 11:40:41,892 ERROR trollmoves.server] Something went wrong during copy of /data/nwcsaf/pps-v2014-ears-out/NWC_data/S_NWC_CT_metopa_62920_20181204T1044000Z_20181204T1107599Z.nc to scp://xvis-m3b/lustre/storeA/project/metproduction/products/satdata_polar/pps-v2014/ears/S_NWC_CT_metopa_62920_20181204T1044000Z_20181204T1107599Z.nc: Channel closed.
[2018-12-04 11:40:41,893 DEBUG trollmoves.server] File "/software/pytroll/lib/python2.7/site-packages/trollmoves-0.1.1-py2.7.egg/trollmoves/server.py", line 579, in move_it
mover(pathname, dest_url, attrs=attrs).copy()
File "/software/pytroll/lib/python2.7/site-packages/trollmoves-0.1.1-py2.7.egg/trollmoves/server.py", line 825, in copy
scp.put(self.origin, self.destination.path)
File "/software/pytroll/lib/python2.7/site-packages/scp-0.10.2-py2.7.egg/scp.py", line 145, in put
self.sanitize(asbytes(remote_path)))
File "/software/pytroll/lib/python2.7/site-packages/paramiko-2.4.1-py2.7.egg/paramiko/channel.py", line 63, in _check
return func(self, *args, **kwds)
File "/software/pytroll/lib/python2.7/site-packages/paramiko-2.4.1-py2.7.egg/paramiko/channel.py", line 241, in exec_command
self._wait_for_event()
File "/software/pytroll/lib/python2.7/site-packages/paramiko-2.4.1-py2.7.egg/paramiko/channel.py", line 1198, in _wait_for_event
raise e

[2018-12-04 11:40:41,893 DEBUG trollmoves.server] Response: pytroll://PPS-v2014/EARS/EXPORT err polarsat@satproc4-a-G3KSDN2 2018-12-04T11:40:41.893261 v1.01 text/ascii Channel closed.

@mraspaud

This comment has been minimized.

Copy link
Member Author

commented Dec 5, 2018

@TAlonglong Did you have a chance to test this ?

@mraspaud mraspaud merged commit bdee6fa into develop Jan 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.