bug: LocalStack Glue Not Working(--exta-jars not found in classpath)

[nilayamkamila]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The application jars are not found in class path for localstack glue, after the glud job run

localstack-main | 2024-05-22T18:42:34.165 DEBUG --- [-functhread7] localstack.utils.run : Executing command: ['bash', '-c', 'echo "MainGateEntry.main(Array())" | AWS_ACCESS_KEY_ID=000000000000 AWS_SECRET_ACCESS_KEY=__internal_call__ AWS_REGION=us-east-1 SPARK_CONF_DIR=/var/lib/localstack/lib/aws-glue-libs/conf java -Dcom.amazonaws.sdk.disableCertChecking=true -cp /var/lib/localstack/lib/gluePyspark:/var/lib/localstack/lib/gluePyspark/*:/var/lib/localstack/lib/aws-glue-libs/conf/:/var/lib/localstack/lib/spark/2.4.3/jars/*:/var/lib/localstack/lib/aws-glue-libs/jarsv1/* -Dscala.usejavacp=true -Xmx1g org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main spark-shell -i /tmp/script-50787a31.scala']

Expected Behavior

The applications jars should be in classpath cp e.g.

localstack-main | 2024-05-22T18:42:34.165 DEBUG --- [-functhread7] localstack.utils.run : Executing command: ['bash', '-c', 'echo "MainGateEntry.main(Array())" | AWS_ACCESS_KEY_ID=000000000000 AWS_SECRET_ACCESS_KEY=__internal_call__ AWS_REGION=us-east-1 SPARK_CONF_DIR=/var/lib/localstack/lib/aws-glue-libs/conf java -Dcom.amazonaws.sdk.disableCertChecking=true -cp /var/lib/localstack/lib/gluePyspark:/var/lib/localstack/lib/gluePyspark/*:/var/lib/localstack/lib/aws-glue-libs/conf/:/var/lib/localstack/lib/spark/2.4.3/jars/*:/var/lib/localstack/lib/aws-glue-libs/jarsv1/*: -Dscala.usejavacp=true -Xmx1g org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main spark-shell -i /tmp/script-50787a31.scala']

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

`services:

localstack:
container_name: "${LOCALSTACK_DOCKER_NAME:-localstack-main}"
image: dockerhub.domain.company.com/localstack/localstack-pro:latest
ports:
- "127.0.0.1:4566:4566" # LocalStack Gateway
- "127.0.0.1:4510-4559:4510-4559" # external services port range
- "127.0.0.1:443:443" # LocalStack HTTPS Gateway (Pro)
environment:
- CURL_CA_BUNDLE=
- DEBUG=${DEBUG:-1}
- GIT_SSL_NO_VERIFY=1
- HTTP_PROXY=http://xxxxxx.com:8099
- HTTPS_PROXY=http://xxxxxx.com:8099
- LOCALSTACK_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxx
- OUTBOUND_HTTP_PROXY=http://xxxxxx.com:8099
- OUTBOUND_HTTPS_PROXY=http://xxxxxx.com:8099
- PERSISTENCE=${PERSISTENCE:-0}
- REQUESTS_CA_BUNDLE=
- SKIP_SSL_CERT_DOWNLOAD=true
volumes:
- "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"`

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

Create a Glue Job with your Jar
awslocal s3 mb s3://XXXXXX-poc && awslocal s3 cp ~/IdeaProjects/XXXXXX-orchXXX/src/main/scala/XXXXXX.scala s3://bankmod-datamigration-poc/data-pipeline/pipeline-artifacts/ && awslocal s3 cp ~/IdeaProjects/XXXXXX/target/classes/XXXXXX-Config.yaml s3://XXXXXX-poc/data-pipeline/pipeline-artifacts/XXXXXX/ && awslocal s3 cp ~/IdeaProjects/data-conduit-orchestrator/target/XXXXXX-1.0-SNAPSHOT.jar s3://XXXXXX-poc/data-pipeline/pipeline-artifacts/XXXXXX/ && awslocal s3 ls s3://XXXXXX-poc

create-job
--name dev-XXXXXX
--role arn:aws:iam::000000000000:role/glue-role
--command '{"Name": "gluestreaming", "ScriptLocation": "s3://XXXXXX-poc/data-pipeline/pipeline-artifacts/XXXXXX/XXXXXX.scala"}'
--default-arguments '{"--job-language": "scala", "--scriptLocation": "s3://XXXXXX-poc/data-pipeline/pipeline-artifacts/XXXXXX/XXXXXX.scala", "--class":"XXXXXX", "--extra-jars": "s3://XXXXXX-poc/data-pipeline/pipeline-artifacts/habanero/XXXXXX-1.0-SNAPSHOT.jar", "--extra-files": "s3://XXXXXX-poc/data-pipeline/pipeline-artifacts/XXXXXX/XXXXXX-Config.yaml"

awslocal glue start-job-run --job-name dev-XXXXXX

Environment

- OS: Mac
- LocalStack: localstack/localstack-pro:latest
  LocalStack version:3.4.0
  LocalStack Docker image sha: Not SHARED
  LocalStack build date:2024-05-22T12:08:27.199717219Z
  LocalStack build git hash:

Anything else?

No response

[localstack-bot]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.