Revert "[Issue 4956]Add default loader for latest pyyaml"

[jerrypeng]

Reverts #4974

Lets revert this for now because its causing Pulsar Function Workers to fail starting up in our integration tests.

2019-10-21T21:03:32.806743794Z + python -V
2019-10-21T21:03:32.809746507Z Python 2.7.16
2019-10-21T21:03:32.979390472Z Traceback (most recent call last):
2019-10-21T21:03:32.979437479Z   File "bin/gen-yml-from-env.py", line 49, in <module>
2019-10-21T21:03:32.979445879Z     conf = yaml.load(open(conf_filename), Loader=yaml.FullLoader)
2019-10-21T21:03:32.979452110Z AttributeError: 'module' object has no attribute 'FullLoader'

This seems to be a problem for python 2.7. Once we move are images to use python 3.7 by default we can bring this change back.

Full log for function worker starting up:

2019-10-21T21:03:32.804486087Z ++ echo
2019-10-21T21:03:32.804626995Z ++ tr '[:upper:]' '[:lower:]'
2019-10-21T21:03:32.806562619Z + FLAG=
2019-10-21T21:03:32.806632205Z + '[' '' = true ']'
2019-10-21T21:03:32.806672570Z + echo 'Python Version ...'
2019-10-21T21:03:32.806723034Z Python Version ...
2019-10-21T21:03:32.806743794Z + python -V
2019-10-21T21:03:32.809746507Z Python 2.7.16
2019-10-21T21:03:32.979390472Z Traceback (most recent call last):
2019-10-21T21:03:32.979437479Z   File "bin/gen-yml-from-env.py", line 49, in <module>
2019-10-21T21:03:32.979445879Z     conf = yaml.load(open(conf_filename), Loader=yaml.FullLoader)
2019-10-21T21:03:32.979452110Z AttributeError: 'module' object has no attribute 'FullLoader'
2019-10-21T21:03:33.378413384Z 2019-10-21 21:03:33,377 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2019-10-21T21:03:33.378506024Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/bookie.conf" during parsing
2019-10-21T21:03:33.378518130Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/broker.conf" during parsing
2019-10-21T21:03:33.378527184Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/functions_worker.conf" during parsing
2019-10-21T21:03:33.378535634Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/global-zk.conf" during parsing
2019-10-21T21:03:33.378544050Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/local-zk.conf" during parsing
2019-10-21T21:03:33.378552477Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/presto_worker.conf" during parsing
2019-10-21T21:03:33.378574774Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/proxy.conf" during parsing
2019-10-21T21:03:33.390787507Z 2019-10-21 21:03:33,390 INFO RPC interface 'supervisor' initialized
2019-10-21T21:03:33.390874023Z 2019-10-21 21:03:33,390 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-10-21T21:03:33.391123089Z 2019-10-21 21:03:33,390 INFO supervisord started with pid 1
2019-10-21T21:03:34.395601299Z 2019-10-21 21:03:34,394 INFO spawned: 'functions-worker' with pid 21
2019-10-21T21:03:35.397666715Z 2019-10-21 21:03:35,397 INFO success: functions-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-10-21T21:04:40.473995238Z 2019-10-21 21:04:40,473 INFO exited: functions-worker (exit status 255; not expected)
2019-10-21T21:04:41.478742777Z 2019-10-21 21:04:41,478 INFO spawned: 'functions-worker' with pid 215
2019-10-21T21:04:42.480653712Z 2019-10-21 21:04:42,480 INFO success: functions-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-10-21T21:05:47.522984806Z 2019-10-21 21:05:47,522 INFO exited: functions-worker (exit status 255; not expected)
2019-10-21T21:05:48.527747430Z 2019-10-21 21:05:48,527 INFO spawned: 'functions-worker' with pid 409
2019-10-21T21:05:49.529612725Z 2019-10-21 21:05:49,529 INFO success: functions-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-10-21T21:06:54.316756347Z 2019-10-21 21:06:54,316 INFO exited: functions-worker (exit status 255; not expected)
2019-10-21T21:06:55.321097684Z 2019-10-21 21:06:55,320 INFO spawned: 'functions-worker' with pid 603
2019-10-21T21:06:56.323222911Z 2019-10-21 21:06:56,322 INFO success: functions-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-10-21T21:08:01.154232192Z 2019-10-21 21:08:01,153 INFO exited: functions-worker (exit status 255; not expected)
2019-10-21T21:08:02.158816137Z 2019-10-21 21:08:02,158 INFO spawned: 'functions-worker' with pid 797
2019-10-21T21:08:03.160892553Z 2019-10-21 21:08:03,160 INFO success: functions-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

[jerrypeng]

rerun integration tests
rerun cpp tests

[tuteng]

The pyyaml version here https://github.com/apache/pulsar/blob/master/docker/pulsar/Dockerfile#L48 seems a bit old.
Why not try to install the new pyyaml https://github.com/apache/pulsar/blob/branch-2.4/docker/pulsar/Dockerfile#L61?

pip2.7 install pyyaml==5.1.2
pip3.7 install pyyaml==5.1.2

[jerrypeng]

@tuteng I have tried that but there are problems with installing a more updated version of PyYAML:

root@28f8b6cfc440:/pulsar# pip3.7 install pyyaml==5.1.2
Collecting pyyaml==5.1.2
  Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)
     |████████████████████████████████| 266kB 1.1MB/s 
Building wheels for collected packages: pyyaml
  Building wheel for pyyaml (setup.py) ... done
  Created wheel for pyyaml: filename=PyYAML-5.1.2-cp37-cp37m-linux_x86_64.whl size=44103 sha256=c9d64793aadb42102719829ba39030b121cd514b3139fcefe9e0584eff3b14ab
  Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030
Successfully built pyyaml
Installing collected packages: pyyaml
  Found existing installation: PyYAML 3.13
ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

feel free to try to get this the work. I just wanted to do something really fast to unblock all PR getting merged because of failing integration tests.

[jerrypeng]

rerun integration tests

[tuteng]

@jerrypeng My guess is that python-yaml is the package that caused this problem. instead of installing python-pip, I directly use the following command to package it in Dockerfile. it can be successful and the integration test can also pass.

RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
RUN python2.7 get-pip.py
RUN pip2.7 install kazoo pyyaml==5.1.2
RUN python3.7 get-pip.py
RUN pip3.7 install kazoo pyyaml==5.1.2

root@c4b89a6b7021:/pulsar# pip2.7 list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package                  Version
------------------------ ---------
apache-bookkeeper-client 4.9.2
certifi                  2019.9.11
chardet                  3.0.4
enum34                   1.1.6
fastavro                 0.22.5
futures                  3.3.0
grpcio                   1.24.1
idna                     2.8
kazoo                    2.6.1
pip                      19.3.1
prometheus-client        0.7.1
protobuf                 3.10.0
pulsar-client            2.5.0
pymmh3                   0.0.5
pytz                     2019.3
PyYAML                   5.1.2
ratelimit                2.2.1
requests                 2.22.0
setuptools               40.8.0
six                      1.12.0
urllib3                  1.25.6
wheel                    0.33.6
root@c4b89a6b7021:/pulsar# pip uninstall PyYAML==5.1.2
Uninstalling PyYAML-5.1.2:
  Would remove:
    /usr/local/lib/python3.7/dist-packages/PyYAML-5.1.2.dist-info/*
    /usr/local/lib/python3.7/dist-packages/yaml/*
Proceed (y/n)? y
  Successfully uninstalled PyYAML-5.1.2
root@c4b89a6b7021:/pulsar# pip install PyYAML==5.1.2
Processing /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030/PyYAML-5.1.2-cp37-cp37m-linux_x86_64.whl
Installing collected packages: PyYAML
Successfully installed PyYAML-5.1.2

After the integration test passes, I will try to upgrade to the new version.