You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Instead of moving aroung data, transmitting a processing program is an approach of map reduce.
The processing program bashreduce is relying on is essentially a bash one liner.
But quoting/escaping hell is really painful. (just running simple sed/awk commands isn't comfortable enough)
Somehow a program distribution mechanism desirable.
Basic idea is
Create a wrapper script. This will distribute the entire bashr pipeline definition.
A pipeline definition will contain
aliases or functions which define map/reduce tasks used in the pipeline
how they are connected
(etc, if necessary)
The wrapper script execute map/reduce tasks as defined in the pipeline. In this step the wrapper script and br script will issue ssh command. And they will orchestrate so that the aliases/functions become available before actual execution of the task.
The text was updated successfully, but these errors were encountered:
Since I could come up with a way to make the quoted one liners used in bred consistent (-I option), closing.
Program distribution would become necessary sooner or later, but it's a separate issue.
Instead of moving aroung data, transmitting a processing program is an approach of map reduce.
The processing program bashreduce is relying on is essentially a bash one liner.
But quoting/escaping hell is really painful. (just running simple sed/awk commands isn't comfortable enough)
Somehow a program distribution mechanism desirable.
Basic idea is
The text was updated successfully, but these errors were encountered: