Use botocore and boto3 compatible with AWS Lambda

[dazza-codes]

Partial fix for #833 but it doesn't modify any CI matrix settings.

This PR might get replaced by #835 but this PR is smaller and tries to use the current packaging setup.

[dazza-codes]

I don't use pipenv routinely and it looks like I missed updating a lock file or something in some of the travis builds:

[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches botocore<1.17.50,<1.18.0,==1.17.44,>=1.17.40,>=1.17.49 (from -r /tmp/pipenv2vfd09h5requirements/pipenv-hx71qvoy-constraints.txt (line 4))

I'm more familiar with using poetry for pure-pip projects. I vaguely recall running into a ton of shell alias problems when using pipenv. I started using it after working with rbenv for a long time, but switched to using conda to provide a python version on my laptop (or docker image for CI) and then using poetry to manage the dependencies. Python packaging and packaging solutions are complicated, but the developments for better sem-ver resolutions are good. (While learning scala, I really like the sbt tooling.)

[mxr]

@dazza-codes I'm not super familiar with botocore & aibotocore so perhaps this advice is not helpful but I think the issue is a conflict between the botocore versions in install_requires vs extras_require in addition to a conflict between what awscli~=1.18.121 needs vs what botocore versions are specified. I was able to get pipenv lock to succeed with this additional change

diff --git a/setup.py b/setup.py
index 12b00a7..46fd099 100644
--- a/setup.py
+++ b/setup.py
@@ -9,3 +9,3 @@ install_requires = [
     # pegged to also match items in `extras_require`
-    'botocore~=1.17.40',
+    'botocore~=1.19.0',
     'aiohttp>=3.3.1',
@@ -22,3 +22,3 @@ extras_require = {
     'awscli': ['awscli~=1.18.121'],
-    'boto3': ['boto3~=1.14.40']
+    'boto3': ['botocore~=1.19.0']
 }

Do you think it's worth trying?

[ediskandarov]

@dazza-codes this PR is stale. Dependencies were updated.

[thehesiod]

wow sorry for not noticing this earlier, this has been done in #844

[thehesiod]

closing since already handled in above PR, also I updated CI to use github actions, I think just need to remove travis in another PR

[mxr]

@thehesiod thanks for the fix!

[thehesiod]

it helps when you need it for the company you work for as well =)

[dazza-codes]

The point of this MR was to relax the very specific constraints on the versions that this library depends on. I would drop the optional dependency on awscli entirely because that should never get pip installed (IMO). Trying to satisfy the dependency versions of all the awscli and botocore can sometimes put pip or poetry into a spin to find all the 100s or 1000s of versions available for these packages. Using a very tight pin in aiobotocore on very specific versions of botocore is a bit too safe, unless aiobotocore will be released in lock-step with botocore releases (1000s of releases).

[thehesiod]

I just realized i ignored there ~= change, i think that's a good idea with the full version so ppl can use third party patches from custom servers

[thehesiod]

actually the ~= change is not possible. The current versioning already allows users to have their own override. For example with the current version: 'botocore>=1.22.8,<1.22.9', you can publish a 1.22.8.1 version and it will be picked up and preferred

[dazza-codes]

@thehesiod - I have run into circumstances where botocore has released a change to a private method that breaks aiobotocore, so I understand the concerns about tight pinning and I understand that it may be unfortunate for aiobotocore to patch private methods on botocore. I think the bigger problem is that aiobotocore touches private details in botocore, and therefore we have to throw away any assumptions about sem-ver behavior 😢. So, even if the approach in this PR were adopted, the sem-ver assumptions are invalid anyway. In this regard, although I would hope the ~= version specifier could work, I'm not sure it would and I can understand closing this PR. (Aside - going back to earlier comments elsewhere about event hooks some time ago, I still don't know if the event architecture is a better solution.)

Anyhow, just to add a brief comment here about understanding the ~= operator. In https://www.python.org/dev/peps/pep-0440/#version-specifiers the ~= allows a "compatible" version. For example, something like botocore~=1.22.8 allows any release that is sem-ver "compatible". Since this specifier includes a patch version, it should be equivalent to: >=1.22.8,==1.22.*. Assuming that botocore would not break any substantial methods in a 1.22.* release, this might be OK.

This is based on looking at the details in https://www.python.org/dev/peps/pep-0440/#compatible-release ... e.g.

~= 2.2
>= 2.2, == 2.*

~= 1.4.5
>= 1.4.5, == 1.4.*