Adding args to submit spark step

[vikramsg]

Feature or Bugfix

• Feature

Detail

• Add args to emr.submit_spark_step. This allows submitting scripts to EMR with CLI args.

Relates

• Missing script CLI args for emr.submit_spark_step #1820

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
• Commit ID: 8680a5a
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[malachi-constant]

Looks good, is it possible to add a test for this param?

[vikramsg]

test

The intention of the tests are not very clear to me. For eg. is f"s3://{bucket}/emr/test_docker.py" an actual script? It would help if this was more clear.

[malachi-constant]

@vikramsg Don't worry about the pytest for now, we are getting an error on an EMR unit test though.

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
• Commit ID: 552432b
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[vikramsg]

AWS CodeBuild CI Report

• CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
• Commit ID: 552432b
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Below is what I see. Is this what you are referring to.

But this is surprising. args is Optional. Why does it say that it is required?

        wr.emr.submit_steps(
            cluster_id=cluster_id,
            steps=[
>               wr.emr.build_spark_step(
                    path=f"s3://{bucket}/emr/test_docker.py",
                    docker_image="123456789123.dkr.ecr.us-east-1.amazonaws.com/docker-emr:docker-emr",
                )
            ],
        )
E       TypeError: build_spark_step() missing 1 required positional argument: 'args'

tests/test_emr.py:159: TypeError

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
• Commit ID: 1e1a34e
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[kukushking]

Thanks @vikramsg for the contribution! Please ignore the CB report, there are a couple of flaky tests unrelated to this change failing