Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] use rsync instead of copymitter #197

Open
wants to merge 1 commit into
base: master
from

Conversation

@ericfranz
Copy link
Contributor

commented Mar 8, 2019

No description provided.

@ericfranz ericfranz added the invalid label Mar 27, 2019

@ericfranz

This comment has been minimized.

Copy link
Contributor Author

commented Mar 27, 2019

This failed to solve the problem.

@ericfranz ericfranz closed this Mar 27, 2019

@ericfranz

This comment has been minimized.

@ericfranz ericfranz reopened this Aug 1, 2019

@ericfranz

This comment has been minimized.

Copy link
Contributor Author

commented Aug 1, 2019

Using this branch actually worked. We might want to clean up the comments and add progress display if the rsync module supports that like the copymitter did.

@MorganRodgers
Copy link
Contributor

left a comment

Looks good so far.

@@ -24,7 +24,8 @@
copymitter = require('copymitter'),

This comment has been minimized.

Copy link
@MorganRodgers

MorganRodgers Aug 1, 2019

Contributor

If copymitter is no longer used then remove the require.

});

This comment has been minimized.

Copy link
@MorganRodgers

MorganRodgers Aug 1, 2019

Contributor

TODO: remove the commented out code.

cp.abort();
});

cp.on('progress', function(count) {

This comment has been minimized.

Copy link
@MorganRodgers

MorganRodgers Aug 1, 2019

Contributor

Ideally the progress callback will be re-implemented. The command rsync -a --info=progress2 --progress $SRC $DST produces output like:

neo-dashboard/node_modules/path-parse/.travis.yml
     18,197,451  80%  424.41kB/s    0:00:41 (xfr#3925, ir-chk=1005/6295)

Giving us a source file, and a percent complete. A caveat is that progress2 was introduced "recently": https://serverfault.com/a/441724. I am not sure what Centos 6 has available, but MacOS for example uses rsync 2.6.9; it is available in Centos 7.

This comment has been minimized.

Copy link
@MorganRodgers

MorganRodgers Aug 1, 2019

Contributor

If we proceed with this solution we'll want to update our list of external programs to ensure that rsync is both installed and has a compatible version.

This comment has been minimized.

Copy link
@ericfranz

ericfranz Aug 7, 2019

Author Contributor

FWIW the rsync node module does appear to provide progress events though perahps it is based on later versions of rsync like you have identified above?

@ericfranz ericfranz removed the invalid label Aug 5, 2019

@MorganRodgers

This comment has been minimized.

Copy link
Contributor

commented Aug 5, 2019

The best solution needs to preserve file modes / permissions etc.

@ericfranz

This comment has been minimized.

Copy link
Contributor Author

commented Aug 5, 2019

Good idea to compare stat attributes of file prior to and after a copy using cp

@MorganRodgers

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2019

I just tried upgrading copymitter and a simple yarn upgrade does not work.

@ericfranz

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

I just tried upgrading copymitter and a simple yarn upgrade does not work.

How did you confirm? Do you have access to a Lustre file system? Or you ran into other problems?

@MorganRodgers

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2019

Sorry, by "did not work" I meant to say that the dependency cannot be upgraded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.