[AIRFLOW-3066] Adding support for AWS Batch parameters

[hugoprudente]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR "[AIRFLOW-3066] Add job parameters to AWSbatch Operator"

Description

• Added the parameters for using dynamic commands on Job Execution

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

nosetests -v tests/contrib/operators/test_awsbatch_operator.py
test_check_success_task_not_raises (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator) ... ok
test_check_success_tasks_raises (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator) ... ok
test_check_success_tasks_raises_failed (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator) ... ok
test_check_success_tasks_raises_multiple (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator) ... ok
test_check_success_tasks_raises_pending (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator) ... ok
test_execute_with_failures (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator) ... ok
test_execute_without_failures (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator) ... ok
test_init (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator) ... ok
test_template_fields_overrides (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator) ... ok
test_wait_end_tasks (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.146s

OK
[2018-11-24 00:59:23,032] {settings.py:203} DEBUG - Disposing DB connection pool (PID 12221)

Commits

• My commits all reference Jira issues: [AIRFLOW-3066] Adding support for AWS Batch parameters

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.

Code Quality

• Passes flake8

[Fokko]

Can we use the >> notation here?

[hugoprudente]

Yes, I'll make the update!

[Fokko]

Do we need the if construction here? Can't we just pass None parameters? Or do parameters =self.parameters or {}

[hugoprudente]

The API does not accept None parameters or {}, both triggers KeyError exception.

[esthernamrm]

Really happy this is being added! Would it make sense to name the keyword arg job_parameters to better differentiate it from the params keyword arg?

[boje]

Will this be merge into master any time soon?