web3 upgrade #339

dbfreem · 2022-04-18T02:12:02Z

Here are the changes to upgrade web3.py for issue #308.

One major change was dropping the support for python 3.6 on ethereum etl since web3.py dropped 3.6 support
You may want to take a look at these notes on upgrading from web3.py v4 to v5 to see if anything jumps out at you.
I did still have the infura test falling locally but I think I needed to just dig in a little more and see why my setup wasn't working. If I have time tomorrow I will look at that.
I had to add a few of those mock response files. I copied ones you already had in there and modified them a little.
Also, I did not test the docker file but noticed it was using python3.6 so just changed the image it was pulling from to python3.7 hope that works.

Let me know what you think, happy to tweak this some more if needed.

removed python 3.6

dbfreem · 2022-04-18T02:19:16Z

I see some more failing test so this is WIP still. I will look at this tomorrow some.

dbfreem · 2022-04-19T01:59:02Z

I think I cleared up most of the test failures. All are passing locally except test_export_origin with infura. I think the issue there is my setup so I think this PR is ready for review.

medvedev1088 · 2022-04-23T16:00:09Z

@dbfreem awesome, thanks for the PR. I can see tests for Python 3.7 still fail in Travis https://app.travis-ci.com/github/blockchain-etl/ethereum-etl/jobs/567463269

py37 installdeps: .[dev,streaming]
ERROR: invocation failed (exit code 1), logfile: /home/travis/build/blockchain-etl/ethereum-etl/.tox/py37/log/py37-1.log
================================== log start ===================================
Processing /home/travis/build/blockchain-etl/ethereum-etl
Collecting web3<6,>=5.29 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/e4/68/70d938f42d673e9d8c842c777ad66e2d8acea99c1c1422a524d4f899731b/web3-5.29.0-py3-none-any.whl (500kB)
Collecting eth-utils==1.10 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/99/71/78170b0fdeb7ee6aa24c2206ac2d0d4f097ff0d9ad4ab4f748e04ea73187/eth_utils-1.10.0-py3-none-any.whl
Collecting eth-abi==2.1.1 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/c9/f4/5d8e501d6535e9e9aab6060bdf6890f266af978e182dc569bf31fb1753b6/eth_abi-2.1.1-py3-none-any.whl
Collecting python-dateutil<3,>=2.8.0 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl (247kB)
Collecting click==7.1.2 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl (82kB)
Collecting ethereum-dasm==0.1.4 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/6f/f1/9556efc3b5a80254a9376bfc339aafb33579429c78b03272a9204f90b1a7/ethereum_dasm-0.1.4-py3-none-any.whl (44kB)
Collecting base58 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/4a/45/ec96b29162a402fc4c1c5512d114d7b3787b9d1c2ec241d9568b4816ee23/base58-2.1.1-py3-none-any.whl
Collecting requests (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-py2.py3-none-any.whl (63kB)
Collecting pytest~=4.3.0 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/c8/52/c455e718f19e4fd7126c81b4910ca2431ded6c28b97e36a1a1f5e5ef7247/pytest-4.3.1-py2.py3-none-any.whl (219kB)
Collecting timeout-decorator==0.4.1 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/07/1c/0d9adcb848f1690f3253dcb1c1557b6cf229a93e724977cb83f266cbd0ae/timeout-decorator-0.4.1.tar.gz
Collecting google-cloud-pubsub==2.1.0 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/41/e4/acb1b0b23d46c1b67976f218a263873d4d1aa2ccf077f73aa3af84b24339/google_cloud_pubsub-2.1.0-py2.py3-none-any.whl (177kB)
Collecting google-cloud-storage==1.33.0 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/f2/c4/973a7b35ec7e49e6e3bf1eca33dfab156220fe67596d3540a0c5fc7ab30f/google_cloud_storage-1.33.0-py2.py3-none-any.whl (92kB)
Collecting kafka-python==2.0.2 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/75/68/dcb0db055309f680ab2931a3eeb22d865604b638acf8c914bedf4c1a0c8c/kafka_python-2.0.2-py2.py3-none-any.whl (246kB)
Collecting sqlalchemy==1.4 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/31/71/984fef7a60b2f13be90dd638cc07f550e9bf1be971cddf0887961f44af78/SQLAlchemy-1.4.0-cp37-cp37m-manylinux2010_x86_64.whl (1.5MB)
Collecting pg8000==1.13.2 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/16/32/ae895597e43bc968e0e3e63860e9932b851115457face0d06d7f451b71fc/pg8000-1.13.2-py3-none-any.whl
Collecting libcst==0.3.21 (from ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/da/b8/788385690cf5f2189f4539545b90ff72cce52919a30b313149c521094e46/libcst-0.3.21-py3-none-any.whl (514kB)
Collecting typing-extensions<5,>=3.7.4.1; python_version < "3.8" (from web3<6,>=5.29->ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/75/e1/932e06004039dd670c9d5e1df0cd606bf46e29a28e65d5bb28e894ea29c9/typing_extensions-4.2.0-py3-none-any.whl
Collecting eth-hash[pycryptodome]<1.0.0,>=0.2.0 (from web3<6,>=5.29->ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/76/90/6e982d67ed0b95968e16e6479565e202091d1d1ebecd4338067429c8cad7/eth_hash-0.3.2-py3-none-any.whl
Collecting jsonschema<5,>=3.2.0 (from web3<6,>=5.29->ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/55/b2/2c4af6a97c3f12c6d5a72b41d328c3996e14e1e46701df3fac1ed65119c9/jsonschema-4.4.0-py3-none-any.whl (72kB)
Collecting hexbytes<1.0.0,>=0.1.0 (from web3<6,>=5.29->ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/4b/34/2c6b549bb318a1fcf8b86438cb414340906a30e29e597f7e0bb19464673b/hexbytes-0.2.2-py3-none-any.whl
Collecting eth-account<0.6.0,>=0.5.7 (from web3<6,>=5.29->ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/47/64/1845c1a2551386eadd7b988ef879b762dacc10cddbb520e77443beb89ffc/eth_account-0.5.7-py3-none-any.whl (101kB)
Collecting eth-typing<3.0.0,>=2.0.0 (from web3<6,>=5.29->ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/0a/a0/88e8c0f27b1909ddf9a67b55d5ec490239abdd00f3f5389c09523c07ea25/eth_typing-2.3.0-py3-none-any.whl
Collecting protobuf<4,>=3.10.0 (from web3<6,>=5.29->ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/af/26/cf27940ece6bb8890a67f741eb9da5359b72749f816edf210a28fe01a247/protobuf-3.20.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0MB)
Collecting lru-dict<2.0.0,>=1.1.6 (from web3<6,>=5.29->ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/68/ea/997af58d4e6da019ad825a412f93081d9df67e9dda11cfb026a3d7cd0b6c/lru-dict-1.1.7.tar.gz
Collecting aiohttp<4,>=3.7.4.post0 (from web3<6,>=5.29->ethereum-etl==1.11.0)
  Downloading https://files.pythonhosted.org/packages/dd/fe/80c594d62a7ff07730fd2cfc3a058498087436d8c938243e0610d1928f0e/aiohttp-3.8.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1MB)
Collecting ipfshttpclient==0.8.0a2 (from web3<6,>=5.29->ethereum-etl==1.11.0)
  ERROR: Could not find a version that satisfies the requirement ipfshttpclient==0.8.0a2 (from web3<6,>=5.29->ethereum-etl==1.11.0) (from versions: 0.4.10, 0.4.11, 0.4.12, 0.4.13, 0.4.13.1, 0.4.13.2, 0.6.0, 0.6.0.post1, 0.6.1)
ERROR: No matching distribution found for ipfshttpclient==0.8.0a2 (from web3<6,>=5.29->ethereum-etl==1.11.0)
=================================== log end ====================================
ERROR: could not install deps [.[dev,streaming]]; v = InvocationError("/home/travis/build/blockchain-etl/ethereum-etl/.tox/py37/bin/python -m pip install '.[dev,streaming]'", 1)
___________________________________ summary ____________________________________
ERROR:   py37: could not install deps [.[dev,streaming]]; v = InvocationError("/home/travis/build/blockchain-etl/ethereum-etl/.tox/py37/bin/python -m pip install '.[dev,streaming]'", 1)
/home/travis/.travis/functions: line 611:  3864 Terminated              travis_jigger "${!}" "${timeout}" "${cmd[@]}"
The command "travis_wait tox $TOX_POSARGS" exited with 1.
Done. Your build exited with 1.

Any idea how to fix it?

dbfreem · 2022-04-23T16:06:18Z

So there is a current issue where web3.py really only supports python 3.7.2 and higher so I think we need to set that in setup.py. Instead of setting it to 3.7

Do you have a link to where you saw this in travis so I can verify it is fixed?

I will take a look at this tomorrow.

ethereum/web3.py#2428

medvedev1088 · 2022-04-23T16:07:27Z

I suspect because we're dropping python 3.6 support, I'll need to bump the major version? Not sure what's the best practice here

medvedev1088 · 2022-04-23T16:09:38Z

@dbfreem re Do you have a link to where you saw this in travis so I can verify it is fixed? Here is the link https://app.travis-ci.com/github/blockchain-etl/ethereum-etl/jobs/567463269. If you can't access it try this one instead https://github.com/blockchain-etl/ethereum-etl/pull/339/checks?check_run_id=6075307688. Thanks

dbfreem · 2022-04-24T21:07:39Z

@medvedev1088 take a look at this now. This should be good to go now.

Not sure why I didn't see the Travis integration before on the PR but they all passed now.

medvedev1088 · 2022-05-03T11:52:17Z

Thanks @dbfreem! Will review and test it this week

jerrinot · 2022-05-06T07:50:23Z

Hello, I understood blockchain-etl does not work on Python 3.10 and this PR could make it compatible again.

It's because the old web3 transitively depends on an ancient version of eth-account, which does not have this tiny 4 letters fix merged and apparently Python 3.10 drop support for the old & deprecated collection API.

dbfreem · 2022-05-06T10:13:00Z

Looking deeper at web3.py support for 3.10 it looks like it is officially only supported in the v6 version that is in beta now. Once that moves to an official release ethereum-etl could be upgrade to v6 and at least from a web3.py standpoint be supported on 3.10. I can't speak for other dependencies on ethereum-etl that may or may not support 3.10.

I can create an issue in ethereum-etl to track this for when web3.py releases it's v6

medvedev1088 · 2022-05-06T11:47:40Z

Published v2.0.0 on pypi and DockerHub

upgraded web3,py and eth-abi

64d16f5

removed python 3.6

dbfreem marked this pull request as draft April 18, 2022 02:18

dbfreem marked this pull request as ready for review April 19, 2022 01:57

travis remove 3.6

e511dac

dbfreem added 4 commits April 24, 2022 11:51

set min version to 3.7.2 in setup.py

15ff2a2

trying to setup travix to run in 3.7.2

8a87ba8

testing travix

e63e703

3.7.2 in travis

50925fc

dbfreem mentioned this pull request May 6, 2022

Add Python 3.10 support #344

Open

medvedev1088 merged commit 53c1b59 into blockchain-etl:develop May 6, 2022

charlielewisme mentioned this pull request May 10, 2023

Upgrade composer version blockchain-etl/polygon-etl#157

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

web3 upgrade #339

web3 upgrade #339

dbfreem commented Apr 18, 2022

dbfreem commented Apr 18, 2022

dbfreem commented Apr 19, 2022

medvedev1088 commented Apr 23, 2022

dbfreem commented Apr 23, 2022

medvedev1088 commented Apr 23, 2022

medvedev1088 commented Apr 23, 2022

dbfreem commented Apr 24, 2022

medvedev1088 commented May 3, 2022

jerrinot commented May 6, 2022

dbfreem commented May 6, 2022 •

edited

medvedev1088 commented May 6, 2022

web3 upgrade #339

web3 upgrade #339

Conversation

dbfreem commented Apr 18, 2022

dbfreem commented Apr 18, 2022

dbfreem commented Apr 19, 2022

medvedev1088 commented Apr 23, 2022

dbfreem commented Apr 23, 2022

medvedev1088 commented Apr 23, 2022

medvedev1088 commented Apr 23, 2022

dbfreem commented Apr 24, 2022

medvedev1088 commented May 3, 2022

jerrinot commented May 6, 2022

dbfreem commented May 6, 2022 • edited

medvedev1088 commented May 6, 2022

dbfreem commented May 6, 2022 •

edited