bump botocore to 1.29.76 #999

jakob-keller · 2023-02-22T14:09:25Z

Description of Change

The botocore dependency has not been updated in a while, causing a number of downstream issues and incompatibilities.

Assumptions

Diff between previous and new botocore sources: boto/botocore@1.27.59...1.29.76

Checklist for All Submissions

I have added change info to CHANGES.rst
If this is resolving an issue (needed so future developers can determine if change is still necessary and under what conditions) (can be provided via link to issue with these details):
- Detailed description of issue closes missing client field endpoint_ruleset_resolver in botocore 1.28.0 #976, closes Bump botocore #993, closes Recent AWS SSO Config change are raising InvalidConfigError on old botocore #995, and closes Project dependencies. #996,
- Alternative methods considered (if any)
- How issue is being resolved: follow instructions in CONTRIBUTING.rst
- How issue can be reproduced

Checklist when updating botocore and/or aiohttp versions

I have read and followed CONTRIBUTING.rst
I have updated test_patches.py where/if appropriate (also check if no changes necessary)
I have ensured that the awscli/boto3 versions match the updated botocore version

thehesiod · 2023-02-22T18:40:01Z

see work I started in #974, we need to research new codepath: SSOTokenProvider._refresher

thehesiod · 2023-02-22T18:40:58Z

oh, I see something along those lines, does this PR account for that?

jakob-keller · 2023-02-22T22:07:46Z

oh, I see something along those lines, does this PR account for that?

I just pushed a commit that supports async token refresh. Please note that this is currently untested. Is anyone able to give it a try and/or suggest unit tests?

thehesiod · 2023-02-22T23:20:58Z

aha, @jakob-keller you da man. Ok for stuff like this I typically pulled tests from botocore. You can see some of our adaptations here: https://github.com/aio-libs/aiobotocore/tree/master/tests/boto_tests. where possible keeping the tests as similar as possible

jakob-keller · 2023-02-22T23:22:40Z

aha, @jakob-keller you da man. Ok for stuff like this I typically pulled tests from botocore. You can see some of our adaptations here: https://github.com/aio-libs/aiobotocore/tree/master/tests/boto_tests. where possible keeping the tests as similar as possible

OK, I will give it a shot.

I just discovered another new blocking codepath in regions.py. Will address that, too.

thehesiod · 2023-02-22T23:24:42Z

oops, forgot probably easier doing it in https://github.com/aio-libs/aiobotocore/tree/master/tests/python3.8/boto_tests, no reason to make it 3.6 compatible

jakob-keller · 2023-02-23T01:20:14Z

OK, I addressed the failing unit tests and the second blocking codepath. It's working locally on my machine. Could you trigger CI again, please?

I haven't gotten around to adding new tests yet. Will try to do that in the coming days.

jakob-keller · 2023-02-25T13:01:44Z

@thehesiod Could you trigger CI again, please? I want to make sure existing tests are fine before I add anything new. Thanks!

thehesiod · 2023-02-25T20:02:25Z

@jakob-keller thanks again, sorry for delay, work + fam have been non-stop.

jakob-keller · 2023-02-28T13:08:02Z

OK, I had a look at the unit tests from upstream and added and/or modified those that I believe are relevant.

@thehesiod: Could you please run CI again to check if there are any remaining issues?

jakob-keller · 2023-03-01T19:25:04Z

Well it looks like test coverage is subpar. Is that a hard requirement? I am not terribly familiar with the test suits of this repo nor upstream, which makes me rather doubtful that I can contribute much more without specific guidance. Any input? Thanks!

thehesiod · 2023-03-01T20:44:44Z

I think the tests may not be running because they're not marked moto: https://github.com/aio-libs/aiobotocore/blob/master/tests/test_monitor.py#L6. Right now we only run moto tests: https://github.com/aio-libs/aiobotocore/blob/master/Makefile#L25 you can run the tests locally as well to check

tibbe · 2023-03-02T15:01:52Z

I would love to see this. I'm trying to decide whether to use aioboto3 in my project and it being ~7 months behind boto3 worries me a bit. For example, if there's a critical bug fix to boto3 and it takes long for it to be picked up by aioboto3 that would be a cause for concern.

I of course understand this is open source and people have limited time. Just wanted to share my thoughts of adopting this package as a potential user.

jakob-keller · 2023-03-02T15:14:43Z

I think the tests may not be running because they're not marked moto: https://github.com/aio-libs/aiobotocore/blob/master/tests/test_monitor.py#L6. Right now we only run moto tests: https://github.com/aio-libs/aiobotocore/blob/master/Makefile#L25 you can run the tests locally as well to check

OK, I believe it's ready for another spin. Awaiting CI run...

jakob-keller · 2023-03-02T21:18:02Z

Well, this time, I believe we are pretty close. Anything else I can do to move this forward?

jakob-keller · 2023-03-06T14:06:49Z

@thehesiod What do you think about the state of this PR? Anything else that I can contribute or need to address?

thehesiod · 2023-03-06T17:38:35Z

@jakob-keller I think it's great, just need a sec to look over it

thehesiod · 2023-03-07T02:58:36Z

looking now

aiobotocore/tokens.py

thehesiod · 2023-03-07T03:37:19Z

aiobotocore/session.py

@@ -167,6 +171,7 @@ async def _create_client(
            )
        else:
            credentials = await self.get_credentials()
+        auth_token = self.get_auth_token()


nice work piping this through!

codecov · 2023-03-07T03:58:55Z

Codecov Report

Merging #999 (36940a0) into master (22a1a4a) will decrease coverage by 15.55%.
The diff coverage is 37.86%.

❗ Current head 36940a0 differs from pull request most recent head 3c8ee05. Consider uploading reports for the commit 3c8ee05 to get more accurate results

@@             Coverage Diff             @@
##           master     #999       +/-   ##
===========================================
- Coverage   86.86%   71.32%   -15.55%     
===========================================
  Files          55       55               
  Lines        5391     5287      -104     
===========================================
- Hits         4683     3771      -912     
- Misses        708     1516      +808

Flag	Coverage Δ
unittests	`71.32% <37.86%> (-15.55%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
tests/test_mturk.py	`44.44% <0.00%> (-5.56%)`	⬇️
aiobotocore/tokens.py	`26.13% <26.13%> (ø)`
aiobotocore/signers.py	`54.48% <31.57%> (-40.69%)`	⬇️
aiobotocore/utils.py	`47.22% <32.72%> (-29.45%)`	⬇️
aiobotocore/credentials.py	`80.73% <62.50%> (-8.04%)`	⬇️
aiobotocore/httpchecksum.py	`65.47% <66.66%> (-0.38%)`	⬇️
aiobotocore/client.py	`84.29% <70.58%> (-1.98%)`	⬇️
aiobotocore/args.py	`100.00% <100.00%> (ø)`
aiobotocore/session.py	`77.45% <100.00%> (+0.92%)`	⬆️
tests/boto_tests/test_credentials.py	`96.86% <100.00%> (-0.38%)`	⬇️
... and 35 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

thehesiod · 2023-03-07T03:59:44Z

looks great! Thanks so much for all your hard work! This was one of the more complicated merges. I'll get this out today

thehesiod · 2023-03-07T04:00:10Z

we should point to this PR why bumping is not so simple :)

thehesiod · 2023-03-07T04:02:18Z

I would love to see this. I'm trying to decide whether to use aioboto3 in my project and it being ~7 months behind boto3 worries me a bit. For example, if there's a critical bug fix to boto3 and it takes long for it to be picked up by aioboto3 that would be a cause for concern.

I of course understand this is open source and people have limited time. Just wanted to share my thoughts of adopting this package as a potential user.

ya I have limited time now with kids and start-up life, have any other volunteers for help with getting releases out sooner? This one took longer than expected due to the complications you can see from this PR. They added several new codepaths that needed to be async'ified

jakob-keller · 2023-03-07T06:10:29Z

ya I have limited time now with kids and start-up life

I feel you!

have any other volunteers for help with getting releases out sooner?

Count me in

paked · 2023-03-08T00:37:16Z

Really appreciate the work here guys. Thanks so much!

martindurant · 2023-05-16T13:19:16Z

Is a release planned with this or more recent botocore dependency? s3fs is being hit by the large number of releases botocore has had since the last 2.5.0 dask/dask#10276

jakob-keller · 2023-05-16T18:28:38Z

Is a release planned with this or more recent botocore dependency?

This has been released as aiobotocore 2.5.0 on Mar 7, 2023. Which minimum release of botocore would you require?

martindurant · 2023-05-16T18:30:07Z

Er, enough to convince conda that downgrading boto3 is preferable to downgrading s3fs by several years.

jakob-keller mentioned this pull request Feb 22, 2023

bump #974

Closed

jakob-keller marked this pull request as ready for review February 22, 2023 14:20

This comment was marked as outdated.

Sign in to view

bump to 1.29.76

256bebc

aio-libs deleted a comment from codecov bot Mar 4, 2023

thehesiod reviewed Mar 7, 2023

View reviewed changes

aiobotocore/tokens.py Show resolved Hide resolved

thehesiod reviewed Mar 7, 2023

View reviewed changes

bump date

3c8ee05

thehesiod approved these changes Mar 7, 2023

View reviewed changes

thehesiod merged commit 72b8dd5 into aio-libs:master Mar 7, 2023

jakob-keller deleted the bump-botocore branch March 7, 2023 06:05

alvarobartt mentioned this pull request Mar 7, 2023

feat: add FsspecConnector to easily integrate new connectors with a fsspec implementation available Unstructured-IO/unstructured#318

Merged

dacevedo12 mentioned this pull request Mar 8, 2023

Add aiobotocore instrumentation open-telemetry/opentelemetry-python-contrib#1709

Closed

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bump botocore to 1.29.76 #999

bump botocore to 1.29.76 #999

jakob-keller commented Feb 22, 2023 •

edited

Loading

thehesiod commented Feb 22, 2023

thehesiod commented Feb 22, 2023

This comment was marked as outdated.

jakob-keller commented Feb 22, 2023

thehesiod commented Feb 22, 2023

jakob-keller commented Feb 22, 2023

thehesiod commented Feb 22, 2023

jakob-keller commented Feb 23, 2023

jakob-keller commented Feb 25, 2023

thehesiod commented Feb 25, 2023

jakob-keller commented Feb 28, 2023

jakob-keller commented Mar 1, 2023

thehesiod commented Mar 1, 2023

tibbe commented Mar 2, 2023

jakob-keller commented Mar 2, 2023

jakob-keller commented Mar 2, 2023

jakob-keller commented Mar 6, 2023

thehesiod commented Mar 6, 2023

thehesiod commented Mar 7, 2023

thehesiod Mar 7, 2023

codecov bot commented Mar 7, 2023

thehesiod commented Mar 7, 2023

thehesiod commented Mar 7, 2023

thehesiod commented Mar 7, 2023

jakob-keller commented Mar 7, 2023

paked commented Mar 8, 2023

martindurant commented May 16, 2023

jakob-keller commented May 16, 2023

martindurant commented May 16, 2023

bump botocore to 1.29.76 #999

bump botocore to 1.29.76 #999

Conversation

jakob-keller commented Feb 22, 2023 • edited Loading

Description of Change

Assumptions

Checklist for All Submissions

Checklist when updating botocore and/or aiohttp versions

thehesiod commented Feb 22, 2023

thehesiod commented Feb 22, 2023

This comment was marked as outdated.

jakob-keller commented Feb 22, 2023

thehesiod commented Feb 22, 2023

jakob-keller commented Feb 22, 2023

thehesiod commented Feb 22, 2023

jakob-keller commented Feb 23, 2023

jakob-keller commented Feb 25, 2023

thehesiod commented Feb 25, 2023

jakob-keller commented Feb 28, 2023

jakob-keller commented Mar 1, 2023

thehesiod commented Mar 1, 2023

tibbe commented Mar 2, 2023

jakob-keller commented Mar 2, 2023

jakob-keller commented Mar 2, 2023

jakob-keller commented Mar 6, 2023

thehesiod commented Mar 6, 2023

thehesiod commented Mar 7, 2023

thehesiod Mar 7, 2023

Choose a reason for hiding this comment

codecov bot commented Mar 7, 2023

Codecov Report

thehesiod commented Mar 7, 2023

thehesiod commented Mar 7, 2023

thehesiod commented Mar 7, 2023

jakob-keller commented Mar 7, 2023

paked commented Mar 8, 2023

martindurant commented May 16, 2023

jakob-keller commented May 16, 2023

martindurant commented May 16, 2023

jakob-keller commented Feb 22, 2023 •

edited

Loading