You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.
Is there an existing issue for this?
Current Behavior
The application jars are not found in class path for localstack glue, after the glud job run
localstack-main | 2024-05-22T18:42:34.165 DEBUG --- [-functhread7] localstack.utils.run : Executing command: ['bash', '-c', 'echo "MainGateEntry.main(Array())" | AWS_ACCESS_KEY_ID=000000000000 AWS_SECRET_ACCESS_KEY=__internal_call__ AWS_REGION=us-east-1 SPARK_CONF_DIR=/var/lib/localstack/lib/aws-glue-libs/conf java -Dcom.amazonaws.sdk.disableCertChecking=true -cp /var/lib/localstack/lib/gluePyspark:/var/lib/localstack/lib/gluePyspark/*:/var/lib/localstack/lib/aws-glue-libs/conf/:/var/lib/localstack/lib/spark/2.4.3/jars/*:/var/lib/localstack/lib/aws-glue-libs/jarsv1/* -Dscala.usejavacp=true -Xmx1g org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main spark-shell -i /tmp/script-50787a31.scala']
Expected Behavior
The applications jars should be in classpath cp e.g.
localstack-main | 2024-05-22T18:42:34.165 DEBUG --- [-functhread7] localstack.utils.run : Executing command: ['bash', '-c', 'echo "MainGateEntry.main(Array())" | AWS_ACCESS_KEY_ID=000000000000 AWS_SECRET_ACCESS_KEY=__internal_call__ AWS_REGION=us-east-1 SPARK_CONF_DIR=/var/lib/localstack/lib/aws-glue-libs/conf java -Dcom.amazonaws.sdk.disableCertChecking=true -cp /var/lib/localstack/lib/gluePyspark:/var/lib/localstack/lib/gluePyspark/*:/var/lib/localstack/lib/aws-glue-libs/conf/:/var/lib/localstack/lib/spark/2.4.3/jars/*:/var/lib/localstack/lib/aws-glue-libs/jarsv1/*: -Dscala.usejavacp=true -Xmx1g org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main spark-shell -i /tmp/script-50787a31.scala']
How are you starting LocalStack?
With a docker-compose file
Steps To Reproduce
How are you starting localstack (e.g.,
bin/localstack
command, arguments, ordocker-compose.yml
)localstack:
container_name: "${LOCALSTACK_DOCKER_NAME:-localstack-main}"
image: dockerhub.domain.company.com/localstack/localstack-pro:latest
ports:
- "127.0.0.1:4566:4566" # LocalStack Gateway
- "127.0.0.1:4510-4559:4510-4559" # external services port range
- "127.0.0.1:443:443" # LocalStack HTTPS Gateway (Pro)
environment:
- CURL_CA_BUNDLE=
- DEBUG=${DEBUG:-1}
- GIT_SSL_NO_VERIFY=1
- HTTP_PROXY=http://xxxxxx.com:8099
- HTTPS_PROXY=http://xxxxxx.com:8099
- LOCALSTACK_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxx
- OUTBOUND_HTTP_PROXY=http://xxxxxx.com:8099
- OUTBOUND_HTTPS_PROXY=http://xxxxxx.com:8099
- PERSISTENCE=${PERSISTENCE:-0}
- REQUESTS_CA_BUNDLE=
- SKIP_SSL_CERT_DOWNLOAD=true
volumes:
- "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"`
Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)
Create a Glue Job with your Jar
awslocal s3 mb s3://XXXXXX-poc && awslocal s3 cp ~/IdeaProjects/XXXXXX-orchXXX/src/main/scala/XXXXXX.scala s3://bankmod-datamigration-poc/data-pipeline/pipeline-artifacts/ && awslocal s3 cp ~/IdeaProjects/XXXXXX/target/classes/XXXXXX-Config.yaml s3://XXXXXX-poc/data-pipeline/pipeline-artifacts/XXXXXX/ && awslocal s3 cp ~/IdeaProjects/data-conduit-orchestrator/target/XXXXXX-1.0-SNAPSHOT.jar s3://XXXXXX-poc/data-pipeline/pipeline-artifacts/XXXXXX/ && awslocal s3 ls s3://XXXXXX-poc
create-job
--name dev-XXXXXX
--role arn:aws:iam::000000000000:role/glue-role
--command '{"Name": "gluestreaming", "ScriptLocation": "s3://XXXXXX-poc/data-pipeline/pipeline-artifacts/XXXXXX/XXXXXX.scala"}'
--default-arguments '{"--job-language": "scala", "--scriptLocation": "s3://XXXXXX-poc/data-pipeline/pipeline-artifacts/XXXXXX/XXXXXX.scala", "--class":"XXXXXX", "--extra-jars": "s3://XXXXXX-poc/data-pipeline/pipeline-artifacts/habanero/XXXXXX-1.0-SNAPSHOT.jar", "--extra-files": "s3://XXXXXX-poc/data-pipeline/pipeline-artifacts/XXXXXX/XXXXXX-Config.yaml"
awslocal glue start-job-run --job-name dev-XXXXXX
Environment
Anything else?
No response
The text was updated successfully, but these errors were encountered: