Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create 'program distribution' mechanism #4

Closed
dakusui opened this issue Feb 22, 2015 · 1 comment
Closed

Create 'program distribution' mechanism #4

dakusui opened this issue Feb 22, 2015 · 1 comment

Comments

@dakusui
Copy link
Owner

dakusui commented Feb 22, 2015

Instead of moving aroung data, transmitting a processing program is an approach of map reduce.
The processing program bashreduce is relying on is essentially a bash one liner.
But quoting/escaping hell is really painful. (just running simple sed/awk commands isn't comfortable enough)

Somehow a program distribution mechanism desirable.

Basic idea is

  1. Create a wrapper script. This will distribute the entire bashr pipeline definition.
  2. A pipeline definition will contain
    1. aliases or functions which define map/reduce tasks used in the pipeline
    2. how they are connected
    3. (etc, if necessary)
  3. The wrapper script execute map/reduce tasks as defined in the pipeline. In this step the wrapper script and br script will issue ssh command. And they will orchestrate so that the aliases/functions become available before actual execution of the task.
@dakusui
Copy link
Owner Author

dakusui commented Feb 27, 2015

Since I could come up with a way to make the quoted one liners used in bred consistent (-I option), closing.
Program distribution would become necessary sooner or later, but it's a separate issue.

@dakusui dakusui closed this as completed Feb 27, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant