Add py38 to CI #732

chadwhitacre · 2020-07-02T11:04:22Z

I've got tox barely running under Python 3.8 locally, making a PR so I can start seeing what CI does with it.

(I don't have perms on Trac yet to make a new ticket, so my branch name uses the old mega-ticket 2065.)

Punchlist

get py38 to run to completion
think through tor_requires/i2p_requires
work around pip 19.0.3-related bug: Add py38 to CI #732 (comment)
make a Trac ticket: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3325
fix anything else outside of py38
add ratchet.py and configure tox.ini to work with it
redo the branch & PR
satisfy towncrier

chadwhitacre · 2020-07-02T11:25:42Z

Hrm, hack to work around lack of a txi2p release is causing problems. 🤔

Collecting txi2p@ git+ssh://git@github.com/str4d/txi2p@0611b9a86172cb70d2f5e415a88eee9f230590b3#egg=txi2p from git+ssh://git@github.com/str4d/txi2p@0611b9a86172cb70d2f5e415a88eee9f230590b3#egg=txi2p (from tahoe-lafs==1.14.0.post359)
  Cloning ssh://git@github.com/str4d/txi2p (to revision 0611b9a86172cb70d2f5e415a88eee9f230590b3) to /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-Sgktrp/txi2p
  Running command git clone -q ssh://git@github.com/str4d/txi2p /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-Sgktrp/txi2p
  Warning: Permanently added the RSA host key for IP address '192.30.255.112' to the list of known hosts.
  git@github.com: Permission denied (publickey).
  fatal: Could not read from remote repository.

I'm also not immediately seeing the connection between tox config and CI config ... where's py38 at? 🧐

chadwhitacre · 2020-07-02T11:50:43Z

Here's the first py38 run!

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704285475

chadwhitacre · 2020-07-02T11:58:59Z

TIL towncrier.

Once I get Trac perms I'll create a proper ticket there, and once I have CI working I'll close this PR and open a fresh one from a new branch with the right ticket number and towncrier will be happy and everyone will be happy and we'll all be happy together.

chadwhitacre · 2020-07-02T12:44:25Z

Interesting bug on Travis with pyinstaller when bumping to Xenial, where the system pip is 19.0.3 vs. ancient 9.0.1 on Trusty. Manifests as no such option: --no-use-pep517. That option was added in 19.1, and our pinned pip is 19.1.1.

Working on Trusty: https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704285470
Broken on Xenial: https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704292628

Hypothesis is that somehow invoking as python -m pip under pyinstaller is finding the system pip vs. the tox-installed pip?

chadwhitacre · 2020-07-02T12:46:24Z

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704292633 shows expected behavior for py38 tho:

  File "/home/travis/build/tahoe-lafs/tahoe-lafs/.tox/py38/lib/python3.8/site-packages/nevow/stan.py", line 150
    raise NotImplementedError, "Stan slot instances are not iterable."
                             ^
SyntaxError: invalid syntax

I.e., the port to Python 3 is blocked on https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3244.

exarkun · 2020-07-02T15:03:03Z

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704292633 shows expected behavior for py38 tho:
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/.tox/py38/lib/python3.8/site-packages/nevow/stan.py", line 150
    raise NotImplementedError, "Stan slot instances are not iterable."
                             ^
SyntaxError: invalid syntax
I.e., the port to Python 3 is blocked on https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3244.

Perhaps we could make the tahoe --version step optional/allowed to fail or move it to after the test suite. We suppose it will still be a while before the Nevow dependency is completely removed but I hope there is a sizable chunk of the codebase and test suite which can be ported and run even while the Nevow dependency lingers in the web code. (As I said though, "I hope".)

wearpants · 2020-07-02T20:42:21Z

I hope there is a sizable chunk of the codebase and test suite which can be ported and run even while the Nevow dependency lingers in the web code. (As I said though, "I hope".)

That's going to be essential, as it sounds like Nevow removal is going to be several more weeks...

Per tahoe-lafs#732 (comment)

chadwhitacre · 2020-07-03T12:06:07Z

💃

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704631138

Ran 93 tests in 2.198s
FAILED (expectedFailures=1, failures=1, errors=84, successes=7)

There's a larger conversation to be had about whether we're ripping out i2p support from Tahoe-LAFS or ensuring that foolscap also supports i2p under Python 3. See: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3033

We'll have to address that old py3.5 target at some point, currently just spitting out the depgraph.

Don't have an SSH key set up in CI. 🐭

Xenial (16.04) is the current default at Travis, and it does have 3.8 available: https://docs.travis-ci.com/user/languages/python/#python-versions

Per tahoe-lafs#732 (comment)

chadwhitacre · 2020-07-03T13:27:23Z

Debugging this --no-use-pep517 issue ...

Working on Trusty

[5076] /home/travis/build/tahoe-lafs/tahoe-lafs/.tox$ /home/travis/virtualenv/python2.7.14/bin/python -m virtualenv --no-download --python /home/travis/virtualenv/python2.7.14/bin/python pyinstaller
created virtual environment CPython2.7.14.final.0-64 in 428888ms
  creator CPython2Posix(dest=/home/travis/build/tahoe-lafs/tahoe-lafs/.tox/pyinstaller, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/travis/.local/share/virtualenv)
    added seed packages: pip==20.1.1, setuptools==44.1.1, wheel==0.34.2
  activators BashActivator,CShellActivator,PythonActivator,FishActivator,PowerShellActivator

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704656231#L1764-L1769

Broken on Xenial

[4552] /home/travis/build/tahoe-lafs/tahoe-lafs/.tox$ /home/travis/virtualenv/python2.7.15/bin/python -m virtualenv --no-download --python /home/travis/virtualenv/python2.7.15/bin/python pyinstaller
Already using interpreter /home/travis/virtualenv/python2.7.15/bin/python
Using real prefix '/opt/python/2.7.15'
New python executable in /home/travis/build/tahoe-lafs/tahoe-lafs/.tox/pyinstaller/bin/python
Overwriting /home/travis/build/tahoe-lafs/tahoe-lafs/.tox/pyinstaller/lib/python2.7/distutils/__init__.py with new content
Installing setuptools, pip, wheel...
done.

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704654720#L1504-L1510

chadwhitacre · 2020-07-03T13:35:40Z

Okay, I think I'm on the right track:

https://discuss.python.org/t/how-should-virtualenv-behave-respective-of-seed-packages-e-g-pip-by-default/4146

chadwhitacre · 2020-07-03T14:10:39Z

Broken on Xenial, virtualenv is 16.1.0.
Working on Trusty, virtualenv is 20.0.25.

Different versions of virtualenv have different seeding algorithms: https://discuss.python.org/t/-/4146). CI puts us several layers deep in virtualenv-ception and I didn't fully unravel the whole thing, but starting with a modern virtualenv seems to work around the issue.

exarkun · 2020-07-03T14:40:26Z

.travis.yml

@@ -73,5 +73,8 @@ matrix:
  - os: linux
    python: '3.5'
    env: T=py35
+  - os: linux
+    python: '3.8'
+    env: T=py38


Feel free to drop Python 3.5 from this config, too. That was from an earlier failed porting effort. We don't need to keep trying to support it.

@exarkun

Per @exarkun: tahoe-lafs#732 (comment)

chadwhitacre · 2020-07-03T21:50:07Z

Moving to #734.

Per tahoe-lafs#732 (comment)

@exarkun

Per @exarkun: tahoe-lafs#732 (comment)

Per tahoe-lafs#732 (comment)

@exarkun

Per @exarkun: tahoe-lafs#732 (comment)

@exarkun

Per @exarkun: tahoe-lafs#732 (comment)

@exarkun

Drops the old py35 job, per @exarkun: tahoe-lafs#732 (comment) Also bumps Ubuntu at Travis so we can get 3.8 eventually. I first went to 3.8 before dropping back to 3.6 as our initial target. Trusty on Travis does include 3.6, but since we want 3.8 "pretty soon," and the OS bump ended up being tricky (see below), let's go ahead and keep the OS bump. Xenial (16.04) is the current default at Travis, and it does have 3.8 available: https://docs.travis-ci.com/user/languages/python/#python-versions The tricky bug is that different versions of virtualenv have different seeding algorithms: https://discuss.python.org/t/-/4146). CI puts us several layers deep in virtualenv-ception and I didn't fully unravel the whole thing, but starting with a modern virtualenv seems to work around the issue.

Per tahoe-lafs#732 (comment)

@exarkun

Drops the old py35 job, per @exarkun: tahoe-lafs#732 (comment) Also bumps Ubuntu at Travis so we can get 3.8 eventually. I first went to 3.8 before dropping back to 3.6 as our initial target. Trusty on Travis does include 3.6, but since we want 3.8 "pretty soon," and the OS bump ended up being tricky (see below), let's go ahead and keep the OS bump. Xenial (16.04) is the current default at Travis, and it does have 3.8 available: https://docs.travis-ci.com/user/languages/python/#python-versions The tricky bug is that different versions of virtualenv have different seeding algorithms: https://discuss.python.org/t/-/4146). CI puts us several layers deep in virtualenv-ception and I didn't fully unravel the whole thing, but starting with a modern virtualenv seems to work around the issue.

Per tahoe-lafs#732 (comment)

chadwhitacre force-pushed the 2065.add-py38-to-ci branch from 8382efe to 069f62b Compare July 3, 2020 11:17

chadwhitacre mentioned this pull request Jul 3, 2020

Port humanreadable.py to Python 3 #733

Merged

chadwhitacre added a commit to chadwhitacre/tahoe-lafs that referenced this pull request Jul 3, 2020

Prevent Nevow from blocking us

c93195b

Per tahoe-lafs#732 (comment)

chadwhitacre added 10 commits July 3, 2020 08:13

Minimally add py38 to CI

547985b

Change python_requires in setup.py

e068119

Ignore eliot.log, some testing artifact

cd392cb

Make a crucial relative import explicit

d38fa74

Bump txi2p to unreleased version that supports Py3

aac0a75

There's a larger conversation to be had about whether we're ripping out i2p support from Tahoe-LAFS or ensuring that foolscap also supports i2p under Python 3. See: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3033

Turn on py38 at Travis

88b1948

We'll have to address that old py3.5 target at some point, currently just spitting out the depgraph.

Use Git over HTTPS instead of SSH

78ca076

Don't have an SSH key set up in CI. 🐭

Bump Ubuntu version at Travis so we can get 3.8

8c48af3

Xenial (16.04) is the current default at Travis, and it does have 3.8 available: https://docs.travis-ci.com/user/languages/python/#python-versions

Upgrade foolscap under Python 3

097bf4e

Prevent Nevow from blocking us

6fa40dc

Per tahoe-lafs#732 (comment)

chadwhitacre force-pushed the 2065.add-py38-to-ci branch 2 times, most recently from 425793c to 60e9420 Compare July 3, 2020 13:05

chadwhitacre force-pushed the 2065.add-py38-to-ci branch from f8b50d0 to 3a05790 Compare July 3, 2020 13:45

chadwhitacre force-pushed the 2065.add-py38-to-ci branch 2 times, most recently from 5c500b6 to 3b4bca1 Compare July 3, 2020 14:25

chadwhitacre force-pushed the 2065.add-py38-to-ci branch from 3b4bca1 to 027d058 Compare July 3, 2020 14:36

exarkun reviewed Jul 3, 2020

View reviewed changes

chadwhitacre added 2 commits July 3, 2020 10:43

Drop the old py35 job

dfb34c2

Per @exarkun: tahoe-lafs#732 (comment)

Start ratcheting up passing tests under Python 3

01158f7

chadwhitacre mentioned this pull request Jul 3, 2020

Add py38 to CI #734

Closed

8 tasks

chadwhitacre closed this Jul 3, 2020

chadwhitacre deleted the 2065.add-py38-to-ci branch July 3, 2020 21:50

chadwhitacre added a commit to chadwhitacre/tahoe-lafs that referenced this pull request Jul 13, 2020

Prevent Nevow from blocking us

c335834

Per tahoe-lafs#732 (comment)

chadwhitacre added a commit to chadwhitacre/tahoe-lafs that referenced this pull request Jul 13, 2020

Drop the old py35 job

315f5e8

Per @exarkun: tahoe-lafs#732 (comment)

chadwhitacre added a commit to chadwhitacre/tahoe-lafs that referenced this pull request Jul 13, 2020

Prevent Nevow from blocking us

54bc1a8

Per tahoe-lafs#732 (comment)

chadwhitacre added a commit to chadwhitacre/tahoe-lafs that referenced this pull request Jul 13, 2020

Drop the old py35 job

141d9f7

Per @exarkun: tahoe-lafs#732 (comment)

chadwhitacre added a commit to chadwhitacre/tahoe-lafs that referenced this pull request Jul 13, 2020

Drop the old py35 job

8fab12c

Per @exarkun: tahoe-lafs#732 (comment)

chadwhitacre added a commit to chadwhitacre/tahoe-lafs that referenced this pull request Jul 14, 2020

Prevent Nevow from blocking us

87fa654

Per tahoe-lafs#732 (comment)

chadwhitacre added a commit to chadwhitacre/tahoe-lafs that referenced this pull request Jul 14, 2020

Prevent Nevow from blocking us

b47b4a4

Per tahoe-lafs#732 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add py38 to CI #732

Add py38 to CI #732

chadwhitacre commented Jul 2, 2020 •

edited

chadwhitacre commented Jul 2, 2020 •

edited

chadwhitacre commented Jul 2, 2020

chadwhitacre commented Jul 2, 2020

chadwhitacre commented Jul 2, 2020

chadwhitacre commented Jul 2, 2020

exarkun commented Jul 2, 2020

wearpants commented Jul 2, 2020

chadwhitacre commented Jul 3, 2020

chadwhitacre commented Jul 3, 2020

chadwhitacre commented Jul 3, 2020

chadwhitacre commented Jul 3, 2020

exarkun Jul 3, 2020

chadwhitacre commented Jul 3, 2020

Add py38 to CI #732

Add py38 to CI #732

Conversation

chadwhitacre commented Jul 2, 2020 • edited

Punchlist

chadwhitacre commented Jul 2, 2020 • edited

chadwhitacre commented Jul 2, 2020

chadwhitacre commented Jul 2, 2020

chadwhitacre commented Jul 2, 2020

chadwhitacre commented Jul 2, 2020

exarkun commented Jul 2, 2020

wearpants commented Jul 2, 2020

chadwhitacre commented Jul 3, 2020

chadwhitacre commented Jul 3, 2020

Working on Trusty

Broken on Xenial

chadwhitacre commented Jul 3, 2020

chadwhitacre commented Jul 3, 2020

exarkun Jul 3, 2020

Choose a reason for hiding this comment

chadwhitacre commented Jul 3, 2020

chadwhitacre commented Jul 2, 2020 •

edited

chadwhitacre commented Jul 2, 2020 •

edited