Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.8 migration #47274

Closed
bayandin opened this issue Nov 27, 2019 · 65 comments
Closed

Python 3.8 migration #47274

bayandin opened this issue Nov 27, 2019 · 65 comments
Labels
help wanted Task(s) needing PRs from the community or maintainers outdated PR was locked due to age python Python use is a significant feature of the PR or issue python-3.8-migration

Comments

@bayandin
Copy link
Member

bayandin commented Nov 27, 2019

Homebrew has a lot of formulae that depend on python and migration at one go ended up with lack of disk space on CI agents (#45337), so this time we're trying to do the migration in several PRs.

The first one PR adds python@3.8 (#47273)

Here is the whole (long) list formulae with total stats for 90 days:

- [x] **node** 1022365 (build)
- [ ] **glib** 808412
- [x] **awscli** 195640
- [ ] **vim** 184753
- [ ] **libxml2** 147953
- [ ] **protobuf** 142980 (build)
- [ ] **sphinx-doc** 116268
- [ ] **protobuf@3.7** 97391 (build)
- [ ] **watchman** 97097
- [x] **ansible** 94704
- [ ] **gdk-pixbuf** 91179 (build)
- [ ] **numpy** 89746
- [x] **tbb** 54591
- [x] **azure-cli** 53282
- [ ] **opencv** 45419
- [ ] **mercurial** 43224
- [ ] **libepoxy** 39119 (build)
- [ ] **macvim** 38855
- [ ] **gobject-introspection** 38489
- [x] **pipenv** 37260
- [ ] **meson** 35550
- [x] **httpie** 33875
- [ ] **sip** 33708
- [x] **docker-compose** 32492
- [x] **geos** 30335
- [ ] **gdal** 25331
- [x] **certbot** 24363
- [ ] **glib-networking** 24313 (build)
- [x] **mitmproxy** 23370
- [ ] **pyqt** 23328
- [ ] **py3cairo** 19426
- [x] **ipython** 18523
- [x] **bazel** 17653 (build)
- [x] **jupyterlab** 17274
- [x] **thefuck** 16965
- [ ] **pygobject3** 15334
- [ ] **mpv** 14718
- [x] **node@12** 14475 (build)
- [x] **vapoursynth** 13133
- [ ] **cython** 10919
- [ ] **ghc** 10426 (build)
- [ ] **fontforge** 10392
- [x] **aws-elasticbeanstalk** 10023
- [x] **pgcli** 9121
- [ ] **pre-commit** 8830
- [x] **scipy** 8576
- [x] **yamllint** 7992
- [ ] **boost-python3** 7350
- [ ] **gpgme** 6850 (build)
- [x] **libpeas** 6509
- [x] **glances** 6508
- [x] **conan** 6481
- [x] **binwalk** 6331
- [ ] **iso-codes** 6060 (build)
- [x] **sshuttle** 5879
- [x] **mackup** 5185
- [x] **cfn-lint** 5108
- [ ] **cassandra** 5068
- [x] **itstool** 4802
- [x] **vtk** 4777
- [x] **z3** 4751
- [x] **ykman** 4538
- [ ] **xapian** 4478
- [x] **s3cmd** 4352
- [x] **asciinema** 4180
- [x] **you-get** 4162
- [x] **supervisor** 3936
- [ ] **opencv@3** 3866
- [x] **bind** 3658
- [x] **ansible-lint** 3655
- [x] **python-yq** 3611
- [x] **mycli** 3536
- [ ] **osquery** 3443 (build)
- [ ] **wxpython** 3430
- [ ] **docutils** 3343
- [x] **streamlink** 3340
- [ ] **handbrake** 3104 (build)
- [ ] **weechat** 3011
- [ ] **pybind11** 2897
- [ ] **flake8** 2877
- [x] **talloc** 2846 (build)
- [ ] **notmuch** 2821
- [ ] **qscintilla2** 2806
- [ ] **afflib** 2677
- [ ] **ocrmypdf** 2551
- [x] **mkdocs** 2538
- [x] **libtensorflow** 2480 (build)
- [x] **magic-wormhole** 2381
- [x] **black** 2357
- [ ] **clingo** 2352
- [ ] **googler** 2350
- [x] **cookiecutter** 2325
- [x] **platformio** 2203
- [ ] **salt** 2117
- [ ] **grc** 2098
- [x] **csvkit** 2094
- [ ] **pyside** 2055
- [x] **pssh** 2034
- [ ] **git-cola** 1991
- [ ] **duplicity** 1943
- [ ] **root** 1757
- [ ] **uhd** 1708
- [ ] **libtorrent-rasterbar** 1690
- [x] **pylint** 1658
- [x] **fonttools** 1622
- [ ] **apache-arrow** 1615
- [ ] **fabric** 1607
- [x] **emscripten** 1549
- [x] **global** 1493
- [ ] **snapcraft** 1474
- [ ] **evince** 1470
- [ ] **gtk-doc** 1423
- [ ] **offlineimap** 1399
- [ ] **unoconv** 1329
- [x] **aws-shell** 1303
- [x] **awslogs** 1259
- [ ] **xdot** 1252
- [ ] **protobuf@3.6** 1211
- [x] **tox** 1206
- [ ] **mesa** 1184 (build)
- [x] **stormssh** 1128
- [x] **gitup** 1126
- [x] **xonsh** 1121
- [x] **bzt** 1105
- [x] **git-review** 1102
- [x] **distcc** 1076
- [ ] **ghc@8.6** 1067 (build)
- [ ] **openimageio** 1045
- [ ] **grip** 1031
- [ ] **mypy** 1016
- [ ] **uwsgi** 1009
- [x] **esptool** 982
- [ ] **gst-python** 957
- [ ] **graph-tool** 940
- [x] **pygments** 938
- [ ] **opencolorio** 920
- [x] **jrnl** 916
- [ ] **dvc** 906
- [ ] **hashpump** 880
- [ ] **varnish** 879 (build)
- [ ] **pipx** 873
- [x] **fpp** 859
- [ ] **eye-d3** 845
- [ ] **glslang** 844 (build)
- [ ] **filebeat** 794 (build)
- [ ] **open-babel** 786
- [x] **rtv** 779
- [ ] **libdazzle** 777 (build)
- [ ] **doitlive** 770
- [x] **legit** 769
- [ ] **autopep8** 753
- [ ] **bear** 745
- [x] **svtplay-dl** 740
- [ ] **theharvester** 734
- [x] **watson** 659
- [ ] **sslyze** 650 -- doesn't support python 3.8 yet https://github.com/nabla-c0d3/sslyze/issues/408
- [ ] **mps-youtube** 623
- [x] **twine-pypi** 614
- [x] **howdoi** 602
- [ ] **ccm** 597
- [ ] **diffoscope** 584
- [ ] **gdcm** 557
- [ ] **pyinstaller** 556
- [ ] **aws-google-auth** 532
- [ ] **gcovr** 525
- [ ] **baobab** 525 (build)
- [ ] **ssh-audit** 523
- [ ] **recon-ng** 521
- [ ] **buku** 511
- [x] **yapf** 498
- [ ] **dnstwist** 464
- [ ] **credstash** 454
- [ ] **ddgr** 421
- [ ] **cmark** 403 (build)
- [ ] **gr-osmosdr** 398
- [ ] **yle-dl** 381
- [x] **jinja2-cli** 377
- [x] **conjure-up** 377
- [ ] **ponysay** 366
- [ ] **suricata** 354
- [ ] **bento4** 353
- [ ] **anime-downloader** 343
- [ ] **spoof-mac** 340
- [ ] **termtosvg** 335
- [ ] **termius** 330
- [x] **trash-cli** 324
- [x] **proselint** 322
- [x] **translate-toolkit** 321
- [ ] **libvoikko** 301 (build)
- [x] **docker-squash** 301
- [ ] **gprof2dot** 291
- [ ] **eralchemy** 283
- [ ] **awsume** 278
- [ ] **shyaml** 277
- [ ] **vit** 272
- [ ] **mat2** 270
- [ ] **nyx** 248
- [ ] **keepassc** 241
- [ ] **mapserver** 239
- [x] **diceware** 238
- [ ] **ghex** 228
- [x] **remarshal** 226
- [x] **mdv** 224
- [ ] **astrometry-net** 218
- [ ] **easy-tag** 217 (build)
- [x] **bandcamp-dl** 216
- [ ] **gcab** 214 (build)
- [x] **gimme-aws-creds** 212 (https://github.com/Homebrew/homebrew-core/pull/49835)
- [ ] **gitfs** 210
- [ ] **urh** 205
- [ ] **gitless** 202
- [ ] **aubio** 201
- [x] **sqlparse** 200
- [x] **juju-wait** 198
- [ ] **b2-tools** 195
- [x] **vsts-cli** 191
- [ ] **snakemake** 188
- [x] **git-plus** 188
- [ ] **yosys** 186
- [ ] **borgmatic** 186
- [x] **pyvim** 185
- [x] **homeassistant-cli** 167
- [ ] **anjuta** 165
- [x] **sceptre** 164
- [ ] **mpi4py** 163
- [ ] **ola** 161
- [ ] **dnsviz** 158
- [ ] **file-roller** 153 (build)
- [ ] **tvnamer** 152
- [ ] **mkvtomp4** 149
- [x] **khard** 148
- [x] **pyinvoke** 147
- [ ] **livestreamer** 147
- [ ] **ispc** 146
- [x] **micropython** 138
- [ ] **meson-internal** 138
- [ ] **robot-framework** 133
- [ ] **bumpversion** 133
- [x] **git-revise** 131
- [ ] **cryptominisat** 129
- [ ] **charm-tools** 124
- [ ] **stone-soup** 122 (build)
- [ ] **gnome-builder** 122 (build)
- [ ] **jsonrpc-glib** 121 (build)
- [ ] **libtorch** 118 (build)
- [ ] **biogeme** 117
- [x] **percol** 115
- [ ] **gexiv2** 113 (build)
- [ ] **alot** 113
- [ ] **template-glib** 111 (build)
- [ ] **poetry** 111
- [x] **kcov** 109 (build)
- [ ] **znc** 104
- [ ] **twarc** 101
- [ ] **nanopb-generator** 101
- [ ] **libgusb** 101 (build)
- [x] **eg-examples** 101
- [ ] **osc** 95
- [ ] **s3ql** 89
- [x] **mongo-orchestration** 84
- [ ] **pympress** 82
- [ ] **wakatime-cli** 80
- [ ] **mm-common** 79
- [ ] **lensfun** 79
- [x] **internetarchive** 76
- [ ] **jenkins-job-builder** 75
- [ ] **aurora-cli** 75
- [ ] **simple-scan** 74 (build)
- [ ] **restview** 73
- [ ] **cmark-gfm** 72 (build)
- [ ] **liquidctl** 70
- [ ] **libproxy** 70
- [x] **csvtomd** 70
- [x] **todoman** 68
- [ ] **python-markdown** 68
- [ ] **llnode** 68 (build)
- [x] **codespell** 68
- [ ] **sslmate** 67
- [ ] **vdirsyncer** 66
- [ ] **genometools** 66
- [ ] **khal** 65
- [ ] **gradio** 65
- [ ] **adios2** 65
- [ ] **zurl** 64
- [ ] **cxxtest** 63
- [ ] **carla** 62
- [ ] **honcho** 61
- [x] **dxpy** 60
- [ ] **gnome-recipes** 59 (build)
- [ ] **libtensorflow@1** 57 (build)
- [x] **instalooter** 56
- [x] **codemod** 56
- [ ] **gom** 54 (build)
- [ ] **git-filter-repo** 54
- [x] **zabbix-cli** 50
- [x] **liblouis** 50
- [x] **gandi.cli** 48
- [ ] **weboob** 47
- [ ] **gtranslator** 47 (build)
- [ ] **pytouhou** 46
- [ ] **pastebinit** 46
- [ ] **arcade-learning-environment** 46
- [ ] **lcm** 45
- [ ] **ghc@8.2** 44 (build)
- [x] **flintrock** 44
- [x] **ydcv** 42
- [x] **fdroidserver** 42
- [x] **rst-lint** 41
- [ ] **trezor-agent** 39
- [x] **statik** 39
- [x] **nbdime** 39
- [ ] **breezy** 39
- [x] **scour** 38
- [x] **choose** 37
- [ ] **euler-py** 34
- [x] **pygitup** 32
- [ ] **graphene** 32 (build)
- [ ] **redex** 31
- [ ] **mftrace** 31
- [ ] **heartbeat** 30 (build)
- [ ] **simgrid** 28
- [ ] **passpie** 28
- [ ] **nicovideo-dl** 28
- [ ] **gucharmap** 27 (build)
- [ ] **libgweather** 25 (build)
- [ ] **ford** 24
- [ ] **onnxruntime** 21 (build)
- [x] **twoping** 19
- [ ] **peru** 16
- [ ] **fobis** 16
- [x] **atdtool** 16
- [ ] **fb-client** 14
- [ ] **mitie** 13
- [ ] **pius** 12
- [ ] **otf2** 12
- [x] **keepkey-agent** 11
- [x] **fades** 11
- [ ] **notifiers** 9
- [ ] **whatmp3** 8
- [x] **twtxt** 8
- [ ] **fbi-servefiles** 5
- [ ] **ansible@2.8** 4
- [ ] **tarsnapper** 3
- [ ] **goolabs** 1
@issyl0 issyl0 mentioned this issue Dec 1, 2019
5 tasks
issyl0 added a commit to issyl0/homebrew-core that referenced this issue Dec 1, 2019
- According to [its GitHub repo](https://github.com/rtomayko/shocco),
  this software is no longer maintained as of 2017.
- It's had 4 installs over the past 90 days, and 19 over the past 365
  days.
- It third from the bottom of our [list of formulae that need building with Python
  3.8](Homebrew#47274).
issyl0 added a commit that referenced this issue Dec 1, 2019
- According to [its GitHub repo](https://github.com/rtomayko/shocco),
  this software is no longer maintained as of 2017.
- It's had 4 installs over the past 90 days, and 19 over the past 365
  days.
- It third from the bottom of our [list of formulae that need building with Python
  3.8](#47274).
@basictheprogram
Copy link
Contributor

To help out follow How To Open a Homebrew Pull Request (and get it merged)?

brew bump-formula-pr

One of the formulas above?

Change

  depends_on "python"
  depends_on "python3"

Test

 brew tests
 brew install --build-from-source <CHANGED_FORMULA>
 brew test <CHANGED_FORMULA>
 brew audit --strict <CHANGED_FORMULA>

Issue PR?

basictheprogram added a commit to basictheprogram/homebrew-core that referenced this issue Dec 2, 2019
WIP Homebrew#45337

Change dependency to python3
@bayandin
Copy link
Member Author

bayandin commented Dec 3, 2019

Hey @basictheprogram,

Currently, we're working on adding python@3.8 and migrating a big bunch of connected to each other formulae in #47326. When we get it merged then we'll be able to migrate all the other formulae which are not linked together so tight, so we'll be able to do it in parallel.

I'll let you know when #47326 is merged and you could help us there. Thanks a lot!

@henryiii
Copy link
Contributor

henryiii commented Jan 9, 2020

I would like to migrate a formula that has:

py_exe = Utils.popen_read("which python3").strip
py_prefix = Utils.popen_read("python3 -c 'import sys;print(sys.prefix)'").chomp
py_inc = Utils.popen_read("python3 -c 'from distutils import sysconfig;print(sysconfig.get_python_inc(True))'").chomp
...
system "python3" ...

I think these would need to be updated to pick up the correct python3.8? And would something like this help:

Formula["python@3.8"].opt_libexec

Edit: Here's what I would propose:

py_exe = Formula["python@3.8"].opt_bin / "python3"
py_prefix = Utils.popen_read("#{py_exe} -c 'import sys;print(sys.prefix)'").chomp
py_inc = Utils.popen_read("#{py_exe} -c 'from distutils import sysconfig;print(sysconfig.get_python_inc(True))'").chomp
...
system py_exe ...

Would that be right? Am I overthinking this, does it somehow have the correct path setup to pick up 3.8 by default?

(I won't be able to migrate it yet, waiting on Numpy)

@bayandin
Copy link
Member Author

bayandin commented Jan 9, 2020

I would like to migrate a formula that has:

py_exe = Utils.popen_read("which python3").strip
py_prefix = Utils.popen_read("python3 -c 'import sys;print(sys.prefix)'").chomp
py_inc = Utils.popen_read("python3 -c 'from distutils import sysconfig;print(sysconfig.get_python_inc(True))'").chomp
...
system "python3" ...

If you're talking about root then it wiil be done in #47326. We can't migrate it separately because it depends on depending on python formulae (brew deps --include-build --tree root) so we can't guarantee that there is no conflict between python and python@3.8 formulae.
For example we already got a conflict for gdcm in that PR.

If about another formula, then (generally) it's not required to do anything extra because for installation we use superenv which includes all dependency (you can check conversation here: #47326 (comment)).

@henryiii
Copy link
Contributor

henryiii commented Jan 9, 2020

Thanks! Yes, primarily for ROOT.

@Bo98
Copy link
Member

Bo98 commented May 15, 2020

brew uses --include-build --include-test python is the best way to get an up-to-date list.

@iMichka
Copy link
Member

iMichka commented May 15, 2020

It was updated; but I probably lost track of some stuff.

Here is the new list:

@max-ae
Copy link
Contributor

max-ae commented May 17, 2020

brew uses --include-build --include-test python does not list gts, but brew deps gts lists the old python formula as a dependency, not python@3.8. So gts should probably be added to the list.

Output $ brew deps gts
gdbm
gettext
glib
jasper
jpeg
libffi
libpng
libtiff
netpbm
openssl@1.1
pcre
**python**
readline
sqlite
xz

@Bo98
Copy link
Member

Bo98 commented May 17, 2020

$ brew deps gts
gdbm
gettext
glib
jasper
jpeg
libffi
libpng
libtiff
netpbm
openssl@1.1
pcre
python@3.8
readline
sqlite
xz

@max-ae
Copy link
Contributor

max-ae commented May 17, 2020

Strange. Forced reinstallation of gts solved this, thanks!

@flatiron32
Copy link

Thanks for the hard work here. Some feedback from a user who lost more time than he likes because this work was hidden to me. My vimrc stopped loading correctly because powerline-status was not found by the python used by vim. I had triple checked that I had not corrupted something in the setup before checking something was not wrong with python3 in vim. This is when I discovered that it was a different version than what installed by the python formula.
When I upgrade formulae I review the caveats and I did not see any warning about vim upgrading the version of python being used. But I might have missed that.

@iMichka
Copy link
Member

iMichka commented May 18, 2020

Sorry if we broke some stuff along the way. There was no specific warning about this. It's quite hard for us to make sure everything survives an update, especially plugins and internal stuff which is not installed by us. We can provide support on a best effort for these cases, but rely mostly on pull requests to improve these kind of migrations.

@iMichka iMichka mentioned this issue May 18, 2020
5 tasks
@henryiii
Copy link
Contributor

henryiii commented May 18, 2020

Is it possible to add the "alt-install" style links, python3.8 and friends, to the python@3.8 formula? Sort of an alt-install like link to /usr/local. It seems like it would much easier to fix things (like adding black into my environment so macvim works again) if I could just do python3.8 -m pip install black instead of brew info python@3.8 followed by copy and pastes of the full paths of things.

Note: in this case, it looks like black has a brew formula. Though I do use this all the time:

/usr/local/opt/python@3.8/bin/python3 -m venv .env

dbogdanov added a commit to MTG/homebrew-essentia that referenced this issue May 28, 2020
Homebrew is migrating formulas to python@3.8
Homebrew/homebrew-core#47274
Our homebrew dependencies have already migrated as well.

Also, properly install the six python package.
@Amar1729
Copy link
Contributor

@iMichka looks like all the formulae in homebrew that support Python 3.8 have been converted, leaving salt, ansible@2.8, bzt, aurora-cli, protobuf@3.7, jupyterlab, and gradio. Is there a draft PR or discussion somewhere concerning homebrew-core's upgrading python to python3.8, or should a new PR be opened for it?

I suppose this draft is outdated -
#54912

@iMichka
Copy link
Member

iMichka commented Jun 17, 2020

#54912 is the latest step to get this done. The remaining formulae are stuck on 3.7 but this should not be an issue. We have just trouble with our migration strategy that we need to iron out, before deploying this change to everybody. Help is welcome to fix the last issue in #54912.

@SMillerDev
Copy link
Member

Salt was just migrated with their last release.

@alexchandel
Copy link
Contributor

Silently converting python to python@3.8 without an intervening uninstall has left thousands of broken symlinks to now-removed python files installed by pip3.7 in /usr/local/bin.

@iMichka
Copy link
Member

iMichka commented Jul 9, 2020

We just released a new version of brew with improvements to the upgrade process. Can you upgrade again and see if things get sorted out?

@henryiii
Copy link
Contributor

henryiii commented Jul 9, 2020

I did a brew upgrade, and then pre-commit broke again, I had to reinstall with brew. (That happened on the 3.7 -> 3.8 change, too, yesterday)

@iMichka
Copy link
Member

iMichka commented Jul 9, 2020

pre-commit installed through pip or brew?

@henryiii
Copy link
Contributor

henryiii commented Jul 9, 2020

Brew always.

@henryiii
Copy link
Contributor

henryiii commented Jul 9, 2020

It's possible it would have been fine if I went from the 3.7 to the newest version without reinstalling it after the initial 3.8 upgrade. I tried Jupyter lab and black just now, and those work. Also, pre-commit has a tendency to break on upgrades, IIRC - I'm pretty sure I've brew reinstalled it a few times.

@iMichka
Copy link
Member

iMichka commented Jul 9, 2020

If you re-installed it between yesterday and today there might have been some issues. Especially if you re-installed with a from-source build. I could not reproduce the error on my side. If a uninstall/install fixed it for you, then we are all good. Migrations are tricky and we are trying to do our best. The python 3.7 to 3.8 was especially difficult for us this time.

@Bo98
Copy link
Member

Bo98 commented Jul 9, 2020

left thousands of broken symlinks to now-removed python files installed by pip3.7 in /usr/local/bin.

If you first upgraded prior to brew 2.4.5's release, run brew cleanup now (after a brew update) and the sylinks which are Homebrew's responsibility should be gone.

@iMichka
Copy link
Member

iMichka commented Jul 9, 2020

Closing here, migration is done.

27 Nov 2019 - 9 July 2020

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
help wanted Task(s) needing PRs from the community or maintainers outdated PR was locked due to age python Python use is a significant feature of the PR or issue python-3.8-migration
Projects
None yet
Development

No branches or pull requests