[Python] Drop python 3.6

[asfimport]

If we are following NEP 29 we can safely drop python 3.6 (released in 2016) and numpy 1.16 support (released in 2019), they got unsupported in January by numpy.

Python 3.6 will reach end of life in 6-7 months anyways, so it's a good target for removal in pyarrow 5.0.0 or 6.0.0. 

Reporter: Adam Lippai / @alippai

Related issues:

• [Python] Drop Numpy 1.16 support (is related to)

PRs and other links:

• GitHub Pull Request #12026

_{Note: This issue was originally created as ARROW-12706. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
@jorisvandenbossche What do you think?

[asfimport]

Adam Lippai / @alippai:
Some context:

• In this field Dask and Pandas dropped the Python 3.6 support
• Python 3.8.x is now in "security bugfixes only" stage, no further regular bugfixes are expected (I'm sharing this because it was really surprising to me) 

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Personally, I am in favor. But I think this needs to be brought to the mailing list for wider feedback (I know last time for dropping Python 3 / a Python version, there were some reservations).

[asfimport]

Antoine Pitrou / @pitrou:
cc @amol-

[asfimport]

Antoine Pitrou / @pitrou:
ML discussion at https://lists.apache.org/thread.html/r159ed8c820d157f4919dc24825681fe47306fb58fa4646f8d8e74d78%40%3Cdev.arrow.apache.org%3E

[asfimport]

Antoine Pitrou / @pitrou:
@emkornfield What is the current status at Google on this?

[asfimport]

Antoine Pitrou / @pitrou:
setuptools_scm is now broken on Python 3.6 because of a transitive dependency:

Traceback (most recent call last):
  File "setup.py", line 632, in <module>
    url='https://arrow.apache.org/'
  File "/opt/conda/envs/arrow/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/opt/conda/envs/arrow/lib/python3.6/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/opt/conda/envs/arrow/lib/python3.6/site-packages/setuptools/dist.py", line 457, in __init__
    for k, v in attrs.items()
  File "/opt/conda/envs/arrow/lib/python3.6/distutils/dist.py", line 281, in __init__
    self.finalize_options()
  File "/opt/conda/envs/arrow/lib/python3.6/site-packages/setuptools/dist.py", line 831, in finalize_options
    ep(self)
  File "/opt/conda/envs/arrow/lib/python3.6/site-packages/setuptools_scm/integration.py", line 90, in infer_version
    config = Configuration.from_file(dist_name=dist_name)
  File "/opt/conda/envs/arrow/lib/python3.6/site-packages/setuptools_scm/config.py", line 181, in from_file
    defn = _load_toml(data)
  File "/opt/conda/envs/arrow/lib/python3.6/site-packages/setuptools_scm/config.py", line 53, in _lazy_tomli_load
    from tomli import loads
  File "/opt/conda/envs/arrow/lib/python3.6/site-packages/tomli/__init__.py", line 6, in <module>
    from tomli._parser import TOMLDecodeError, load, loads
  File "/opt/conda/envs/arrow/lib/python3.6/site-packages/tomli/_parser.py", line 1
    from __future__ import annotations
    ^
SyntaxError: future feature annotations is not defined

See https://github.com/apache/arrow/runs/4521199401?check_suite_focus=true

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
We can pin tomli to 1.x for now (tomli 2.0 dropped Python 3.6 support, https://github.com/hukkin/tomli/blob/master/CHANGELOG.md#200), if setuptools_scm doesn't do that themselves (they still claim python_requires = >=3.6)

[asfimport]

Antoine Pitrou / @pitrou:
We can, but that suggests we might soon have to pin other transitive dependencies because of similar issues.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I am not sure that will necessarily be the case. Our other setup dependencies are I think cython, numpy and setuptools. And for example numpy already dropped Python 3.6 before.

[asfimport]

Micah Kornfield / @emkornfield:
My understanding is that most Google have committed to following the end of life policy set upstream by python maintainers.  IIUC, I think this means we can drop python 3.6 for next release if this is accurate https://endoflife.date/python

 

 

[asfimport]

Antoine Pitrou / @pitrou:
@kszucs Do you want to take this?

[asfimport]

Antoine Pitrou / @pitrou:
I submitted a PR to drop Python 3.6 support but what we should do about Numpy 1.16?

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 12026
#12026

[asfimport]

Antoine Pitrou / @pitrou:
Created ARROW-15214 for Numpy 1.16.