Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Output 'java.io.IOException: Argument list too long' using by sh operator. #768

Closed
wakamen opened this Issue Apr 25, 2018 · 4 comments

Comments

Projects
None yet
2 participants
@wakamen
Copy link

wakamen commented Apr 25, 2018

Hello.
This workflow is failed.
Does digdag pass environment variable automatically when using the sh operator?

sample code

  • set_heavy_list_variable.dig
+set_some_variables:
  _export:
    rb:
      require: variable_setter.rb
  rb>: VariableSetter.set_heavy

+exec_shell_command:
  sh>: echo hello
  • variable_setter.rb
class VariableSetter
  def set_heavy
    Digdag.env.store(nums: 1_000_000.times.map { |_i| 1 })
  end
end
  • execution log
$ digdag run set_heavy_list_variable.dig 
2018-04-25 15:36:04 +0900: Digdag v0.9.24
2018-04-25 15:36:07 +0900 [WARN] (main): Using a new session time 2018-04-25T00:00:00+00:00.
2018-04-25 15:36:07 +0900 [INFO] (main): Using session /Users/xxx/dev/samples/digdag/.digdag/status/20180425T000000+0000.
2018-04-25 15:36:07 +0900 [INFO] (main): Starting a new session project id=1 workflow name=set_heavy_list_variable session_time=2018-04-25T00:00:00+00:00
2018-04-25 15:36:08 +0900 [INFO] (0018@[0:default]+set_heavy_list_variable+set_some_variables): rb>: VariableSetter.set_heavy
2018-04-25 15:36:16 +0900 [INFO] (0018@[0:default]+set_heavy_list_variable+exec_shell_command): sh>: echo hello
2018-04-25 15:36:16 +0900 [ERROR] (0018@[0:default]+set_heavy_list_variable+exec_shell_command): Task failed with unexpected error: java.io.IOException: Cannot run program "/bin/sh" (in directory "/Users/xxx/dev/samples/digdag"): error=7, Argument list too long
java.lang.RuntimeException: java.io.IOException: Cannot run program "/bin/sh" (in directory "/Users/xxx/dev/samples/digdag"): error=7, Argument list too long
	at com.google.common.base.Throwables.propagate(Throwables.java:160)
	at io.digdag.standards.operator.ShOperatorFactory$ShOperator.runTask(ShOperatorFactory.java:139)
	at io.digdag.util.BaseOperator.run(BaseOperator.java:35)
	at io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:312)
	at io.digdag.cli.Run$OperatorManagerWithSkip.callExecutor(Run.java:694)
	at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:254)
	at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:137)
	at io.digdag.core.agent.LocalWorkspaceManager.withExtractedArchive(LocalWorkspaceManager.java:25)
	at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:135)
	at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:119)
	at io.digdag.cli.Run$OperatorManagerWithSkip.run(Run.java:676)
	at io.digdag.core.agent.MultiThreadAgent.lambda$null$0(MultiThreadAgent.java:127)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Cannot run program "/bin/sh" (in directory "/Users/xxx/dev/samples/digdag"): error=7, Argument list too long
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at io.digdag.standards.command.SimpleCommandExecutor.start(SimpleCommandExecutor.java:22)
	at io.digdag.standards.command.DockerCommandExecutor.start(DockerCommandExecutor.java:56)
	at io.digdag.standards.operator.ShOperatorFactory$ShOperator.runTask(ShOperatorFactory.java:126)
	... 15 common frames omitted
Caused by: java.io.IOException: error=7, Argument list too long
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 18 common frames omitted
2018-04-25 15:36:16 +0900 [INFO] (0018@[0:default]+set_heavy_list_variable^failure-alert): type: notify
error: 
  * +set_heavy_list_variable+exec_shell_command:
    Cannot run program "/bin/sh" (in directory "/Users/xxx/dev/samples/digdag"): error=7, Argument list too long (runtime)

Task state is saved at /Users/xxx/dev/samples/digdag/.digdag/status/20180425T000000+0000 directory.
  * Use --session <daily | hourly | "yyyy-MM-dd[ HH:mm:ss]"> to not reuse the last session time.
  * Use --rerun, --start +NAME, or --goal +NAME argument to rerun skipped tasks.

my environment

$ digdag --version
0.9.24

$ java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)
@hiroyuki-sato

This comment has been minimized.

Copy link
Contributor

hiroyuki-sato commented Apr 25, 2018

Hello, @wakamen

The environment variables which defined in rb> pass sh> automatically.

You can check environment variable easily with env command.

+set_some_variables:
  _export:
    rb:
      require: variable_setter.rb
  rb>: VariableSetter.set_heavy

+exec_shell_command:
  sh>: env
#  sh>: echo hello
class VariableSetter
  def set_heavy
#    Digdag.env.store(nums: 1_000_000.times.map { |_i| 1 })
    Digdag.env.store(nums: 10.times.map { |_i| 1 })
  end
end
2018-04-25 17:00:11 +0900 [INFO] (0017@[0:default]+set_heavy_list_variable+exec_shell_command): sh>: env
...
nums=[1,1,1,1,1,1,1,1,1,1]

I think your issue is /bin/sh limitation, not digdag.
sh doesn't allow to set long environment value like nums=[1,1,1,1,1.......] .

You can reproduce it without Digdag.

ruby -e 'puts "export TEST=" + "x" * 10000000'  > /tmp/test.txt
. /tmp/test.text
ls
-bash: /bin/ls: Argument list too long

You also can't set many environment variable too.

#!/bin/sh

#for i in {1..1000000} ; do
for i in {1..50000} ; do
  echo $i
  export TEST$i="$i"
done
#env
exec /bin/sh -c 'echo "test"'
test.sh: line 9: /bin/sh: Argument list too long
test.sh: line 9: /bin/sh: Undefined error: 0
@wakamen

This comment has been minimized.

Copy link
Author

wakamen commented Apr 25, 2018

@hiroyuki-sato Thank you for reply!

I see.
I found that environment variable of digdag is using system environment variable.
So it was an error by sh operator due to bash limitation.

Thank you very much.

@hiroyuki-sato

This comment has been minimized.

Copy link
Contributor

hiroyuki-sato commented Apr 25, 2018

Hello, @wakamen
Could you close this issue when you solve it?

@wakamen

This comment has been minimized.

Copy link
Author

wakamen commented Apr 25, 2018

@hiroyuki-sato Thank you. solved. I close the issue.

@wakamen wakamen closed this Apr 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.