Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix local fs support #355

Merged
merged 18 commits into from
May 9, 2019
Merged

Fix local fs support #355

merged 18 commits into from
May 9, 2019

Conversation

simleo
Copy link
Member

@simleo simleo commented May 8, 2019

This PR contains multiple fixes for the hadoop-on-local-fs scenario, i.e., the one where fs.defaultFS is set to file:///, and adds a new Travis job to test it.

Main highlights:

  • Hadoop's pipes submitter is still broken due to MAPREDUCE-4000. However, this PR applies the required fix to our own submitter, so Pydoop now supports running mapreduce applications on the local fs via pydoop submit.
  • The above issue further stresses the fact that the "official" way to run applications is via pydoop submit. Examples still relying on PipesRunner have been rewritten to use pydoop submit, so that they can run correctly on the local fs. Running via mapred pipes is still supported when it works, and we're checking that in int_test/mapred_submitter (which serves as a sanity check for our submitter).
  • PipesRunner and related stuff can now be dropped, since they would just be a manteinance burden.

@simleo simleo merged commit 0247d48 into crs4:develop May 9, 2019
@simleo simleo deleted the test_local_fs branch May 9, 2019 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant