-
Notifications
You must be signed in to change notification settings - Fork 587
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
automatically tarball directories? #23
Comments
I have been running a (somewhat) specialized implementation of this feature that might be generalized for this purpose. Is this still an open issue in MRJob? |
(whoops, didn't mean to close that. wooo GitHub keyboard shortcuts) Yes, this is still an open issue. We didn't do this because we didn't have a good use case. Would love to see your code. |
I'm currently archiving the specified files on runner instantiation and appending the resulting file(s) onto the python_archives list. Although I'd generally prefer to avoid mucking with the caller's passed-in values, this is an easy shortcut. Thoughts? |
Sorry for the slow response. Just to be totally clear, can you show me what your command line looks like and/or some sample code? |
Probably better to |
Finally going to do this! Just going to create tarballs (since even old versions of Hadoop support them) and not do any filtering of hidden files. |
auto-archive directories (fixes #23)
make unit tests pass on Travis CI
We have a python_archives option which allows you to upload a tarball and stick it in the $PYTHONPATH. It seems kind of silly, but it would probably be helpful to people if we would automatically tar up directories for them.
We probably want to automatically remove stray editor/MacFuse crud (~, .#, ._*) like we do when bootstrapping mrjob.
Not going to do this until someone asks for it. :)
The text was updated successfully, but these errors were encountered: