Make the library compatible with pre-installed libraries on Glue 4.0

[FredStober]

Is your idea related to a problem? Please describe.
Glue 4.0 has the following pre-installed:
numpy==1.23.4
pyarrow==10.0.0

The following PRs changed requirements for the latest version of awswrangler:
PR #1731 updated the dependencies to allow to satisfy the pyarrow requirement
PR #1831 updated the dependencies that prevent to satisfy the numpy requirement

Describe the solution you'd like
Fix the dependencies, so awswrangler can be installed with minimal changes (ideally no downgrades / upgrades of pre-installed packages) on glue 4.0.

[kukushking]

Thanks for opening this @FredStober, I think it should be <=1.23.4:

numpy = [
    { version = ">=1.21.0 <=1.23.4", markers = "python_full_version >= '3.7.1' and python_full_version < '3.11.0'" },
    { version = "^1.23.4", markers = "python_full_version >= '3.11.0'" },
]

Will open a PR with a fix

[jmahlik]

@kukushking

I'm not sure if #1831 would fix this for us.

On python 3.9.

    our-package-with-50-other-pinned-deps-and-pinned-numpy x.x.x depends on numpy==1.23.5
    awswrangler 2.18.0 depends on numpy<1.23.4 and >=1.21.0; python_full_version >= "3.7.1" and python_full_version < "3.11.0"

An ideal situation for us at least would be something like one of these:

# The ultimate ideal case, users can pin on their end
numpy = [
    { version = ">=1.21.0", markers = "python_full_version >= '3.7.1' and python_full_version < '3.11.0'" },
    { version = ">=1.23.4", markers = "python_full_version >= '3.11.0'" },
]

# A reasonably high value like 2.0.0 or a few minor versions out. 

numpy = [
    { version = ">=1.21.0,<2.0.0", markers = "python_full_version >= '3.7.1' and python_full_version < '3.11.0'" },
    { version = ">=1.23.4,<2.0.0", markers = "python_full_version >= '3.11.0'" },
]

# Maybe the next "minor-ish" version since numpy sometimes introduces deprecations in minors, but they are really unlikely to break anything used via wrangler.
# At least no patch version in an upper bound please :) then we can't get bug fixes downstream
# But this will cause an issue when 1.25.0 comes out, and we'll have to wait until the next release cycle of wrangler to upgrade numpy or do weird tricks to install it separately. 

numpy = [
    { version = ">=1.21.0 <1.24.0", markers = "python_full_version >= '3.7.1' and python_full_version < '3.11.0'" },
    { version = ">=1.23.4,<1.24.0", markers = "python_full_version >= '3.11.0'" },
]

This might be a separate/bigger issue to discuss. Wrangler seems to be the one dependency causing grief when updating packages due to pretty restrictive upper bounds. I understand restricting upper bounds somewhat given the general user base might not be pinning deps or have a test suite to catch breakage.

If the upper bounds could be a little less restrictive in general, that would save us a lot of back and fourth. I can open that separately if desired.