New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adam-submit Error: Number of executors must be a positive number on EMR 5.13.0/Spark 2.3.0 #1991

Closed
heuermh opened this Issue May 24, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@heuermh
Copy link
Member

heuermh commented May 24, 2018

$ adam-submit --version
Using ADAM_MAIN=org.bdgenomics.adam.cli.ADAMMain
Using spark-submit=/usr/lib/spark/bin/spark-submit
Error: Number of executors must be a positive number

Something seems off with the default value for numExecutors

$ adam-submit --master yarn  --num-executors 2 --
Using ADAM_MAIN=org.bdgenomics.adam.cli.ADAMMain
Using spark-submit=/usr/lib/spark/bin/spark-submit
18/05/24 15:52:03 INFO ADAMMain: ADAM invoked with args:

       e        888~-_         e            e    e
      d8b       888   \       d8b          d8b  d8b
     /Y88b      888    |     /Y88b        d888bdY88b
    /  Y88b     888    |    /  Y88b      / Y88Y Y888b
   /____Y88b    888   /    /____Y88b    /   YY   Y888b
  /      Y88b   888_-~    /      Y88b  /          Y888b

Usage: adam-submit [<spark-args> --] <adam-args>

...

$ adam-submit --master yarn --conf spark.executor.instances=2 --
Using ADAM_MAIN=org.bdgenomics.adam.cli.ADAMMain
Using spark-submit=/usr/lib/spark/bin/spark-submit
18/05/24 15:52:25 INFO ADAMMain: ADAM invoked with args:

       e        888~-_         e            e    e
      d8b       888   \       d8b          d8b  d8b
     /Y88b      888    |     /Y88b        d888bdY88b
    /  Y88b     888    |    /  Y88b      / Y88Y Y888b
   /____Y88b    888   /    /____Y88b    /   YY   Y888b
  /      Y88b   888_-~    /      Y88b  /          Y888b

Usage: adam-submit [<spark-args> --] <adam-args>
...
@fnothaft

This comment has been minimized.

Copy link
Member

fnothaft commented Jun 13, 2018

This isn't an ADAM issue, no? This would be a Spark issue.

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Jun 14, 2018

spark-submit worked fine in that environment, I'll have to test locally with 2.3.0 and 2.3.1 to see if I can replicate.

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Jun 23, 2018

Cannot replicate anywhere but on EMR, there must be something off with their Spark installation or default Spark configuration.

@heuermh heuermh closed this Jun 23, 2018

@heuermh heuermh added this to the 0.24.1 milestone Jun 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment