Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AIRFLOW-6730] Use total_seconds instead of seconds #7363

Merged
merged 3 commits into from Feb 28, 2020
Merged

[AIRFLOW-6730] Use total_seconds instead of seconds #7363

merged 3 commits into from Feb 28, 2020

Conversation

saguziel
Copy link
Contributor

@saguziel saguziel commented Feb 5, 2020 •
edited

Issue link: AIRFLOW-6730

Make sure to mark the boxes below before creating PR: [x]

  • Description above provides context of the change
  • Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID*
  • Unit tests coverage for changes (not needed for documentation changes)
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

* For document-only changes commit message can start with [AIRFLOW-XXXX].

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

@boring-cyborg boring-cyborg bot added area:Scheduler Scheduler or dag parsing Issues k8s provider:google Google (including GCP) related issues labels Feb 5, 2020
@ashb
Copy link
Member

ashb commented Feb 5, 2020

🤦‍♂️ Guess we've just been lucky and the default timeouts don't fire.

ashb
ashb reviewed Feb 5, 2020
View reviewed changes
airflow/providers/google/cloud/operators/postgres_to_gcs.py Outdated
@@ -94,7 +94,7 @@ def convert_type(self, value, schema_type):
return datetime.timedelta(
hours=formated_time.tm_hour,
minutes=formated_time.tm_min,
seconds=formated_time.tm_sec).seconds
seconds=formated_time.tm_sec).total_seconds()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is I think wrong.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

In [7]: timedelta(hours=12, minutes=30, seconds=30).seconds                                                                                                                          
Out[7]: 45030

In [8]: timedelta(hours=12, minutes=30, seconds=30).total_seconds()                                                                                                                  
Out[8]: 45030.0

(total_seconds returns a float, previously this was an int. I don't think that is intended.)

ashb
ashb requested changes Feb 5, 2020
View reviewed changes
Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One change that is wrong

@ashb
Copy link
Member

ashb commented Feb 6, 2020

Ah - the "native" representation of a timedelta is (days, seconds) so it's only giving the wrong value when the delta is >= 1 day. So nothing is likely to break as a result of this, but using d.total_seconds() is correct (or int(d.total_seconds())) in some places.

@saguziel saguziel requested a review from ashb February 11, 2020 00:58
ashb
ashb reviewed Feb 13, 2020
View reviewed changes
tests/providers/google/cloud/operators/test_dataproc.py Outdated
@@ -104,7 +104,7 @@
"autoscaling_config": {"policy_uri": "autoscaling_policy"},
"config_bucket": "storage_bucket",
"initialization_actions": [
{"executable_file": "init_actions_uris", "execution_timeout": "600s"}
{"executable_file": "init_actions_uris", "execution_timeout": "600.0s"}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is my only question -- does DataProc accept this, or does it only accept whole-integer seconds here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be int:
https://developers.google.com/protocol-buffers/docs/reference/google.protobuf?&_ga=2.173161123.14377043.1581598687-968688242.1573564810#duration

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://developers.google.com/protocol-buffers/docs/proto3#json

@saguziel WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the python typing indicates float, but I think that is just within the airflow project.

I will just cast it to int since that is guaranteed to not break anything

@saguziel
Copy link
Contributor Author

This passed on my travis-ci so i will merge

@saguziel saguziel merged commit 008b4ba into apache:master Feb 28, 2020
galuszkak pushed a commit to FlyrInc/apache-airflow that referenced this pull request Mar 5, 2020
@saguziel @galuszkak
[AIRFLOW-6730] Use total_seconds instead of seconds (apache#7363)
6c52f42 
* [AIRFLOW-6730] Use total_seconds instead of seconds

* adds tests and fixes types issue

* fix test
kaxil pushed a commit that referenced this pull request Mar 31, 2020
@saguziel @kaxil
[AIRFLOW-6730] Use total_seconds instead of seconds (#7363)
7ab0416 
* [AIRFLOW-6730] Use total_seconds instead of seconds

* adds tests and fixes types issue

* fix test

(cherry picked from commit 008b4ba)
kaxil pushed a commit that referenced this pull request Mar 31, 2020
@saguziel @kaxil
[AIRFLOW-6730] Use total_seconds instead of seconds (#7363)
2315fb9 
* [AIRFLOW-6730] Use total_seconds instead of seconds

* adds tests and fixes types issue

* fix test

(cherry picked from commit 008b4ba)
kaxil pushed a commit that referenced this pull request Mar 31, 2020
@saguziel @kaxil
[AIRFLOW-6730] Use total_seconds instead of seconds (#7363)
637b0b9 
* [AIRFLOW-6730] Use total_seconds instead of seconds

* adds tests and fixes types issue

* fix test

(cherry picked from commit 008b4ba)
kaxil pushed a commit that referenced this pull request Apr 1, 2020
@saguziel @kaxil
[AIRFLOW-6730] Use total_seconds instead of seconds (#7363)
b48655e 
* [AIRFLOW-6730] Use total_seconds instead of seconds

* adds tests and fixes types issue

* fix test

(cherry picked from commit 008b4ba)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaxil kaxil kaxil left review comments

@turbaszek turbaszek turbaszek left review comments

@ashb ashb ashb approved these changes

Assignees
No one assigned
Labels
area:Scheduler Scheduler or dag parsing Issues provider:google Google (including GCP) related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@saguziel @ashb @kaxil @turbaszek