Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

local mode splits .gz files #333

Closed
coyotemarin opened this issue Jan 11, 2012 · 2 comments
Closed

local mode splits .gz files #333

coyotemarin opened this issue Jan 11, 2012 · 2 comments
Labels
Bug
Milestone
v0.3.2

Comments

@coyotemarin
Copy link
Collaborator

In an attempt to emulate Hadoop, local mode splits *.gz files, which isn't actually correct (or possible, even, without unarchiving the whole).

We should stop doing this, as Yelp has tests that depend on this special treatment of *.gz files.
@coyotemarin coyotemarin mentioned this issue Jan 11, 2012
Closed
@ghost ghost assigned irskep Jan 11, 2012
@irskep
Copy link
Contributor

irskep commented Jan 12, 2012

What about *.bz2?

@irskep irskep closed this as completed Jan 12, 2012
@irskep irskep reopened this Jan 12, 2012
@coyotemarin
Copy link
Collaborator Author

I did a bit more research, it looks like there is a patch in later version of Hadoop to allow splitting of .bz2 files. So I guess local mode should continue to split *.bz2 files because it'll catch more bugs that way.

irskep added a commit that referenced this issue Jan 12, 2012
@irskep
Merge pull request #335 from irskep/dont_split_gz_files_333
8828b9d 
Don't split compressed files (#333)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug
Projects
None yet
Milestone
v0.3.2
Development

No branches or pull requests
2 participants
@irskep @coyotemarin