Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Accessing files from HDFS #18

Closed
ramanbedi1989 opened this Issue Aug 28, 2013 · 2 comments

Comments

Projects
None yet
2 participants

Now I am stuck with a problem that our program uses yaml files for loading data. The program cannot read files from local file system while running in hadoop.
I can put my files into HDFS.
But is there a way in wukong-hadoop to read files from HDFS in ruby program?

Contributor

dhruvbansal commented Sep 24, 2013

One common approach is to use a deploy pack (
http://github.com/infochimps-labs/wukong-deploy) to hold all your Wukong
(or other code) AND your YAML configuration files.

You can either

  1. place this deploy pack at the same location on all Hadoop workers
  2. place this deploy pack on an NFS mount accessible to all Hadoop workers.

Both methods should ensure that your Hadoop map/reduce tasks have access to
the code (and config files) at runtime.

-dhruv

On Wed, Aug 28, 2013 at 7:40 AM, Raman Bedi notifications@github.comwrote:

Now I am stuck with a problem that our program uses yaml files for loading
data. The program cannot read files from local file system while running in
hadoop.
I can put my files into HDFS.
But is there a way in wukong-hadoop to read files from HDFS in ruby
program?


Reply to this email directly or view it on GitHubhttps://github.com/infochimps-labs/wukong/issues/18
.

Thanks your comment was really helpful. Thanks a ton.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment