[AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow

[OElesin]

Make sure you have checked all steps below.

JIRA

• My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-2310
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a JIRA issue.

Description

• Here are some details about my PR, including screenshots of any UI changes:
  Currently, there is no integration with AWS Glue Jobs. My pull request is basically an improvement to integrate running AWS Glue jobs with Airflow.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
  Added tests.contrib.test_aws_glue_job_hook.py.
  However, moto the test class for boto3 does not support AWS Glue mock currently

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

• Passes git diff upstream/master -u -- "*.py" | flake8 --diff

[codecov-io]

Codecov Report

Merging #3504 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3504   +/-   ##
=======================================
  Coverage   77.14%   77.14%           
=======================================
  Files         203      203           
  Lines       15123    15123           
=======================================
  Hits        11667    11667           
  Misses       3456     3456

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0f4d681...253171f. Read the comment docs.

[Fokko]

Thanks @OElesin
Can you squash your commits into one single commit?

[OElesin]

@Fokko, kindly review.

Thanks

[Fokko]

@OElesin something must have gone wrong with the rebasing/merging, I see a lot of unrelated changes.

[OElesin]

@Fokko, this was the same problem I emphasized with the previous PR. Can you please point out the unrelated changes?

[Fokko]

It isn't clean. If you look at the Files tab, you see a lot of files changed related to GCP. That doesn't make sense.

[suma-ps]

@OElesin Do you plan to resolve the merge issues soon? Looking forward to using the Glue operator soon, thanks!

[cnidus]

@OElesin I'm also really interested in this submission.

[OElesin]

@suma-ps and @cnidus ,

I have plans of working fixing this. But I kinda get squashing commits mixed up at times. I would try to resolve this by weekend and resubmit the PR.

Thanks @ALL

[xoen]

Just "accidentally" got here while looking at something else but I thought it was worth commenting on this loop.

Isn't this a potentially infinite loop? Once in the loop, how are these 3 variables updated? I don't see the call to get the up-to-date status after the time.sleep().

[oelesinsc24]

Well, it's not an infinite loop as it exits the loop once the job status is in any of these states:
FAILED, STOPPED or SUCCEEDED.

[xoen]

Sorry @oelesinsc24 just saw this comment. I'm probably missing something but I can't see where the failed, stopped, completed variables change value inside the while loop. The else branch only sleeps, doesn't this mean that the if condition will always be false? 🤷‍♂

[sid88in]

@OElesin AWS Glue Operator will be a great addition! Let's fix this PR and get this out?

[simonvanderveldt]

@OElesin Have you been able to make any progress on this?
There's one duplicate file in there airflow/contrib/aws_glue_job_hook.py vs airflow/contrib/hook/aws_glue_job_hook.py.

[edit] From looking at the PR it seems it will both create a job (using boto3's create_job()) as well as run the job (using boto3's start_job_run()). IMHO at the very least these should be separate things so one can run jobs without creating them.
To expand on this, we create the jobs using a normal development workflow/CI and we just want to be able to run them from Airflow. IMHO it's not Airflow's responsibility to create resources, it should only invoke them.

FYI All the unrelated changes are because you merged master into your branch instead of rebasing your branch on top of master

[oelesinsc24]

@OElesin Have you been able to make any progress on this?
There's one duplicate file in there airflow/contrib/aws_glue_job_hook.py vs airflow/contrib/hook/aws_glue_job_hook.py.

[edit] From looking at the PR it seems it will both create a job (using boto3's create_job()) as well as run the job (using boto3's start_job_run()). IMHO at the very least these should be separate things so one can run jobs without creating them.
To expand on this, we create the jobs using a normal development workflow/CI and we just want to be able to run them from Airflow. IMHO it's not Airflow's responsibility to create resources, it should only invoke them.

FYI All the unrelated changes are because you merged master into your branch instead of rebasing your branch on top of master

This is absolutely correct. And also makes the implementation way easier. I would make the necessary changes and add the commit.

[oelesinsc24]

Everyone, I have moved this PR to: #4068. I will close this later and ask that we track changes on the new PR.

Apologies for the inconveniences.

[Fokko]

Moved to #4068