[SPARK-29672][BUILD][PYTHON][FOLLOW-UP] Recover PySpark via pip installation with deprecated Python 2, 3.4 and 3.5

[HyukjinKwon]

What changes were proposed in this pull request?

The RC fails to install against Python 2.7 via pip. We deprecated but didn't remove Python 2, 3.4 and 3.5 support yet. This PR partially reverts the changes from SPARK-29672 to recover Python 2, 3.4 and 3.5 pip installation.

python2.7 -m pip install https://dist.apache.org/repos/dist/dev/spark/v3.0.0-rc1-bin/pyspark-3.0.0.tar.gz

...
Collecting https://dist.apache.org/repos/dist/dev/spark/v3.0.0-rc1-bin/pyspark-3.0.0.tar.gz
  Using cached https://dist.apache.org/repos/dist/dev/spark/v3.0.0-rc1-bin/pyspark-3.0.0.tar.gz (203.0 MB)
    ERROR: Command errored out with exit status 1:
     command: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/_1/bzcp960d0hlb988k90654z2w0000gp/T/pip-req-build-sfCnmZ/setup.py'"'"'; __file__='"'"'/private/var/folders/_1/bzcp960d0hlb988k90654z2w0000gp/T/pip-req-build-sfCnmZ/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/_1/bzcp960d0hlb988k90654z2w0000gp/T/pip-req-build-sfCnmZ/pip-egg-info
         cwd: /private/var/folders/_1/bzcp960d0hlb988k90654z2w0000gp/T/pip-req-build-sfCnmZ/
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/_1/bzcp960d0hlb988k90654z2w0000gp/T/pip-req-build-sfCnmZ/setup.py", line 27
        file=sys.stderr)
            ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Why are the changes needed?

To keep the deprecated support instead of removing.

Does this PR introduce any user-facing change?

No, it's the change in unreleased branches only yet.

How was this patch tested?

./build/mvn -DskipTests -Phive -Phive-thriftserver clean package
cd python
python2.7 setup.py sdist
python2.7 -m pip install dist/pyspark-3.1.0.dev0.tar.gz

[SparkQA]

Test build #121408 has finished for PR 28243 at commit 4323b56.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

@dongjoon-hyun, @viirya, @shaneknapp can you take a quick look please?

[nchammas]

LGTM.

[nchammas]

Maybe convert this into a warning that the user is using a version of Python that Spark has deprecated support for?

[dongjoon-hyun]

+1 for @nchammas 's comment.

[HyukjinKwon]

Sure, will add.

[HyukjinKwon]

Hm, seems like it's tricky to print out logs during pip installation (see pypa/pip#2933). Let me just merge this one.

[viirya]

hmm, so we only recover setup.py to make it installable, but not all testing stuffs? Is it safe to do?

[HyukjinKwon]

I think it is a more serious issue that we said deprecated but the support was actually dropped. It might be less safe, though. We will drop the support in the master and old Pythons will be the end of support so wont cost a lot to recover this back I guess.

[dongjoon-hyun]

Then, could you switch this PR to branch-3.0 only, @HyukjinKwon ?

Or, do we want to keep this in Apache Spark 3.1.0 which will this winter? If we want to keep this in 3.1.0. I'm also okay to merge to master first.

[HyukjinKwon]

Oh, let me target this to master/branch-3.0 for now and remove it together later when we actually drop Python 2, 3.4 and 3.5, which will likely be very soon .. but you never know before it actually happens :).. it's easier for me to track/remember the changes at least..

[dongjoon-hyun]

Got it. I agree~

[HyukjinKwon]

Merged to master and branch-3.0.

Thanks all!