python@3.10: migrate aliases #87075
python@3.10: migrate aliases #87075
Conversation
BrewTestBot
added
the
automerge-skip
|
Not sure this is ready yet, as |
carlocab
added
the
maintainer feedback
XuehaiPan
force-pushed
the
python3-aliases
branch
from
0c9e4c8
to
048bec3
Compare
|
Sorry, when I said this isn't ready because We should wait until most/all Python-dependent formulae have been migrated to |
I fully understand that. I pushed more commits here and we can merge these changes in a single PR when we are ready (maybe 2 weeks later). |
|
Oh, the review is not requested by me. We are probably not ready for this yet. The migration for |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
The migration process for $ git rev-parse HEAD # current HEAD
ec64d6dd8537b662ecb84a500508a89aa12e77fe
$ for formula in $(find Formula -name 'python@*.rb' | xargs basename -s .rb | sort -rV); do echo "$(grep -rlF 'depends_on "'"${formula}"'"' Formula | wc -l)\tformulae depend on ${formula}"; done
268 formulae depend on python@3.10
335 formulae depend on python@3.9
11 formulae depend on python@3.8
3 formulae depend on python@3.7
$ git checkout a4d29eb6431ac9b57f07d6b29f3c5a1238ef2370
Note: switching to 'a4d29eb6431ac9b57f07d6b29f3c5a1238ef2370'.
HEAD is now at a4d29eb6431 python@3.9: migrate aliases
$ for formula in $(find Formula -name 'python@*.rb' | xargs basename -s .rb | sort -rV); do echo "$(grep -rlF 'depends_on "'"${formula}"'"' Formula | wc -l)\tformulae depend on ${formula}"; done
425 formulae depend on python@3.9
22 formulae depend on python@3.8
5 formulae depend on python@3.7 |
|
I wouldn't count on the remaining migration being completed in three weeks. It might, but I also think it'll take longer than the first batch of formulae that's been migrated, since most of the remaining formulae to be migrated live in a few large dependency trees that need to be migrated together. That will take some work. |
Not that much movement in the past ten days. |
XuehaiPan
force-pushed
the
python3-aliases
branch
from
a03cbbc
to
6ca43aa
Compare
|
Any updates? It has been two months since Python 3.10 was released. |
|
Running the script now gives 10 more formula on 3.10. You can't just wait here, if you want this to happen soon you'll need to help migrate them. |
|
Please add label: python-3.10-migration so that those who want to help get to Python 3.10 know how to discover which formulae to work on. |
|
I think we could open an issue to track the progress of the Python 3.10 migration process. Should we migrate formula-by-formula in a single pull request or migrate bunch of formulae in a single pull request (test them at once)? |
This comment has been minimized.
This comment has been minimized.
|
It's not quite that simple, since dependency trees need to be migrated together. See #87277 |
UPDATE: There are 7 pull requests needed in total:
HEAD: d99ad28
Python code to produce: #!/usr/bin/env python3
# %%
import json
import shutil
from collections import defaultdict, deque
from subprocess import check_output, run, DEVNULL, PIPE
from typing import OrderedDict
from toposort import toposort # pip install toposort
brew = shutil.which('brew')
assert brew is not None, 'Could not find `brew` executable in PATH.'
git = shutil.which('git')
assert git is not None, 'Could not find `git` executable in PATH'
HOMEBREW_CORE_REPOSITORY = check_output([brew, '--repository', 'homebrew/core']).decode().strip()
HEAD = check_output([git, '-C', HOMEBREW_CORE_REPOSITORY, 'rev-parse', 'HEAD']).decode().strip()
print(f'brew: {brew}')
print(f'git: {git}')
print(f'HEAD: {HEAD}')
# %%
def brew_uses(formula):
# Directly call `brew uses ...` is too slow
# Use `grep` command instead
files = run(['grep', '-rlF', f'depends_on "{formula}"'],
cwd=f'{HOMEBREW_CORE_REPOSITORY}/Formula',
stdout=PIPE, stderr=DEVNULL).stdout.decode().split()
return sorted(map(lambda rubyfile: rubyfile[:-3], files))
ALL_FORMULAE = brew_uses('python@3.9')
ALL_FORMULAE.remove('python-tk@3.9')
ALL_FORMULAE_SET = set(ALL_FORMULAE)
USES = dict()
DEPENDENCIES = {formula: set() for formula in ALL_FORMULAE}
for formula in ALL_FORMULAE:
USES[formula] = sorted(set(brew_uses(formula)) & ALL_FORMULAE_SET)
for downstream in USES[formula]:
DEPENDENCIES[downstream].add(formula)
# %%
def flatten(nested):
flattened = []
for item in nested:
if not isinstance(item, str) and hasattr(nested, '__iter__'):
flattened.extend(flatten(item))
else:
flattened.append(item)
return flattened
def topo_order(dependencies):
# Topological sort (in the same level, sort by number of downstream nodes)
return list(map(lambda fl: sorted(fl, key=lambda f: len(USES[f]), reverse=True), toposort(dependencies)))
def topo_depends(formula):
dependencies = defaultdict(set)
q = deque([formula])
q.extend(USES[formula])
q.extend(DEPENDENCIES[formula])
while q:
f = q.popleft()
if f in dependencies:
continue
dependencies[f] = DEPENDENCIES[f]
q.extend(USES[f])
q.extend(DEPENDENCIES[f])
return topo_order(dependencies)
families = []
family_roots = set(topo_order(DEPENDENCIES)[0])
while family_roots:
root = family_roots.pop()
family = topo_depends(root)
family_roots.difference_update(family[0])
families.append(family)
# %%
results = OrderedDict({'stand-alones': []})
for family in sorted(families, key=lambda family: (len(flatten(family)), family[0][0])):
family_flattened = flatten(family)
n_formulae = len(family_flattened)
if len(family_flattened) > 1:
roots = family[0]
results[roots[0]] = family
else:
results['stand-alones'].extend(family_flattened)
results['stand-alones'] = [results['stand-alones']]
# %%
for i, (root, family) in enumerate(results.items(), start=1):
family_flattened = flatten(family)
n_formulae = len(family_flattened)
print()
print(f'- **{i}. Family `{root}` ({n_formulae} {"formulae" if n_formulae > 1 else "formula"})**')
print()
if root == 'stand-alones':
for formula in family_flattened:
print(f' - [ ] `{formula}`')
else:
print(f' ```')
print(f' {family}')
print(f' ```')
# %%
with open('families.json', 'wt') as file:
json.dump(results, file, indent=2) |
|
CI tap syntax has passed. No more conflict in the dependency trees. $ git rev-parse HEAD
40171bc81d27ffbd0c53e5503f57365c4d901670
$ for formula in $(find Formula -name 'python@*.rb' | xargs basename -s .rb | sort -rV); do echo "$(grep -rlF 'depends_on "'"${formula}"'"' Formula | wc -l)\tformulae depend on ${formula}"; done
599 formulae depend on python@3.10
103 formulae depend on python@3.9
9 formulae depend on python@3.8
2 formulae depend on python@3.7 |
Implemented this at #107517. |
Since depends_on "python@3.9"
python = Formula["python@3.9"].bin/"python3"or depends_on "python@3.9"
python = which("python3")There are bunch of formulae need to add If we make a symlink |
|
For those formulae in particular: There aren't too many of those. I count less than 35. But it is not necessary to fix them all in the same PR. Some alternatives:
The second doesn't completely fix the ones that look for Of course, there could be many more formulae that assume [*] This is also useful for the formulae that still have both |
|
Require 91 + 38 + 28 separate PRs to add $ for formula in $(find Formula -name 'python@*.rb' | xargs basename -s .rb | sort -rV); do echo "$(grep -rlF 'Formula["'"${formula}"'"].opt_bin' Formula | wc -l)\tformulae use python3 in ${formula}'s opt_bin"; done
91 formulae use python3 in python@3.10's opt_bin
38 formulae use python3 in python@3.9's opt_bin
0 formulae use python3 in python@3.8's opt_bin
0 formulae use python3 in python@3.7's opt_bin
$ grep -rlF 'which("python3")' Formula | wc -l
28It would be good to handle this in We can also add new attributes to def python3
bin/"python#{version.major_minor}"
end
def python
bin/"python#{version.major_minor}"
endThen in the dependent: -python = Formula["python@3.9"].bin/"python3"
+python = Formula["python@3.9"].python3I think all of these may be out of the scope about migrating alias. Although |
XuehaiPan
force-pushed
the
python3-aliases
branch
from
8a1a03c
to
6ee7709
Compare
XuehaiPan
force-pushed
the
python3-aliases
branch
from
6ee7709
to
51a5652
Compare
This makes sure that Python does not install major-versioned files (e.g. `python3`, `python3-config`) and instead only installs files with both a major and minor version (e.g. `python3.9`, `python3.9-config`). This will allow us to have multiple versions of Python linked into `HOMEBREW_PREFIX` without conflicts. If users want to call `python3.9` as `python3`, then they can add `libexec/"bin"` to their `PATH`. Going forward, this will help simplify migrating to newer versions of Python, since we will not need to make the newer version keg-only while dependent formulae are migrated. See discussion at Homebrew/discussions#2776, and documentation at https://github.com/python/cpython#installing-multiple-versions Closes Homebrew#87075.
|
Thanks for your help with pushing the migration, @XuehaiPan. |
brew install --build-from-source <formula>, where<formula>is the name of the formula you're submitting?brew test <formula>, where<formula>is the name of the formula you're submitting?brew audit --strict <formula>(after doingbrew install --build-from-source <formula>)? If this is a new formula, does it passbrew audit --new <formula>?Migrate aliases (
python,python3,python@3) topython@3.10.