[AIRFLOW-2557] Remove paged test for s3

[bolkedebruin]

Make sure you have checked all steps below.

JIRA

• My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-2557
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a JIRA issue.

Description

• Here are some details about my PR, including screenshots of any UI changes:

Paged tests for s3 are taking over 120 seconds. The
functionality testes is part of the standard boto suite
and part of its own tests.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

tests removed.

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

• Passes git diff upstream/master -u -- "*.py" | flake8 --diff

cc @Fokko @NielsZeilemaker

[NielsZeilemaker]

You could reduce the number from 5000 to 1001, as a single call will return at most 1000 results.

[codecov-io]

Codecov Report

Merging #3455 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3455      +/-   ##
==========================================
- Coverage   76.89%   76.89%   -0.01%     
==========================================
  Files         203      203              
  Lines       15118    15120       +2     
==========================================
+ Hits        11625    11626       +1     
- Misses       3493     3494       +1

Impacted Files	Coverage Δ
airflow/hooks/S3_hook.py	94.69% <100%> (+0.09%)	⬆️
airflow/models.py	87.2% <0%> (-0.05%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 91dd368...a117d1a. Read the comment docs.

[bolkedebruin]

@NielsZeilemaker I know, but I looked at the code and I don’t think that makes sense. We are not testing anything inside airflow by it and the tests together would still take around 20seconds.

[Fokko]

Well, it verifies this logic: https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/S3_hook.py#L98-L102

[bolkedebruin]

No it doesn’t, that part will always be tested as you can see in the test coverage as well. There will be always one page if you have results.

[Fokko]

It looks like the branch is still being hit: https://codecov.io/gh/apache/incubator-airflow/pull/3455/src/airflow/hooks/S3_hook.py#L98...102

🤔

[NielsZeilemaker]

It will, as you always have a single page, but testing to see if multiple pages work would have my preference, as I actually encountered this bug while using the hook. Maybe there is a quicker method to add the files?

[bolkedebruin]

Can you lower the page size somehow?

[bolkedebruin]

@NielsZeilemaker @Fokko updated. Page size is a supported setting

Note: @Fokko lets not accept PRs anymore that go over 2s in tests.

[Fokko]

Thanks @bolkedebruin, looking good. Codecov tells me that the branch is still being hit.

I agree that normal tests should run within 2seconds, maybe drop a line on the dev mailinglist?