Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Command-line (ssh) distributed executor #30

Open
frsyuki opened this issue Jan 29, 2015 · 0 comments
Open

Command-line (ssh) distributed executor #30

frsyuki opened this issue Jan 29, 2015 · 0 comments

Comments

@frsyuki
Copy link
Contributor

frsyuki commented Jan 29, 2015

Idea:
I want to execute Embulk using this command line:

$ embulk run config.yml --execute-command "ssh %(find_host %{index}) embulk run-task %{task} %{index}"

With this command, embulk executes the given ssh command for each task. Then embulk run-task executes the given task on another host.
This is the simplest way to run embulk on distributed environment. We don't have to install Hadoop, YARN, Mesos, etc.

Some distributed execution environment such as mpich2 or Sun Grid Engine may work using this command-line.

See also: #29

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

1 participant