Conversation
boring-cyborg
bot
added
area:providers
area:system-tests
kind:documentation
provider:amazon
There was a problem hiding this comment.
What should happen when both wait_for_completion and deferrable are set to True?
There was a problem hiding this comment.
Good question. The AMPP trigger README suggests placing the deferrable check before wait_for_completion, indicating the preference is to defer. In this case, both options accomplish the same goal - wait for the DataBrew job to reach a terminal status. So, should setting both to True make a difference?
There was a problem hiding this comment.
I'm not sure should we store it into the glue.py or move it into the databrew.py. It somehow related to the Glue, but internally in boto3 it is separate client, and I guess it might have separate AWS API. So in theory both approach could be applicable, that is just question for clarification
There was a problem hiding this comment.
Yeah I agree. I guess both are fine and it is just a question of opinion but I would move it too to another file named databrew.py
There was a problem hiding this comment.
I wasn't sure which way it should really go, it is shown in the console under Glue. The waiter, however, needs to be named databrew, so it probably makes more sense to split it out into its own set of files. I will make the changes, if this is the direction we should go.
There was a problem hiding this comment.
Yeah, it's in an awkward position and I guess either would work, but I'd lean towards separate in this case as well.
There was a problem hiding this comment.
How should this appear in the provider.yaml file? Should it have its own entry named 'Amazon Glue DataBrew', or be listed under the existing 'Glue' (like glue_crawler)?
There was a problem hiding this comment.
There is a simple answer to where to put it.
If I am a user of databrew what is the first place that I will look to find databrew related code? (This question is regardless of Airflow itself.. it's the same for doc search or any other thing related to the service)
obviously I will go under Amazon/aws but what after?
would it be under glue / databrew / glue_databrew ?
If most users are going naturally to A then code should be in A.
This is the only question that matters.
There was a problem hiding this comment.
Yeah I agree. I guess both are fine and it is just a question of opinion but I would move it too to another file named databrew.py
ferruzzi
left a comment
ferruzzi
left a comment
There was a problem hiding this comment.
Left some comments and suggestions
There was a problem hiding this comment.
Yeah, it's in an awkward position and I guess either would work, but I'd lean towards separate in this case as well.
|
Static check failures and unit tests. Could you please take a look? |
I don't think all changes were committed/pushed. Just rebased and pushed all changes. Local tests passed. |
Co-authored-by: Hussein Awala <hussein@awala.fr>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Co-authored-by: Andrey Anshin <Andrey.Anshin@taragol.is>
ephraimbuddy
added
the
changelog:skip
7 participants
closes: #22037
Created Glue DataBrew start job Operator.