Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate engine tools/gn script (and others) to work with Python 3 #83043

Closed
14 tasks done
cbracken opened this issue May 20, 2021 · 5 comments · Fixed by flutter/engine#27835
Closed
14 tasks done

Migrate engine tools/gn script (and others) to work with Python 3 #83043

cbracken opened this issue May 20, 2021 · 5 comments · Fixed by flutter/engine#27835
Assignees
@cbracken
Labels
engine flutter/engine repository. See also e: labels. P3 Issues that are less important to the Flutter project team Infra upgrades, team productivity, code health, technical debt. See also team: labels.

Comments

@cbracken
Copy link
Member

cbracken commented May 20, 2021 •
edited
Loading

Currently, many of the Python scripts in the engine repo, and some of our dependencies, like the Dart SDK, rely on Python 2.7 which was officially end-of-lifed in 2020. We should update our tooling to work with both Python 2.7 and Python 3.

Platform verification:
@cbracken cbracken added team Infra upgrades, team productivity, code health, technical debt. See also team: labels. engine flutter/engine repository. See also e: labels. labels May 20, 2021
@cbracken cbracken self-assigned this May 20, 2021
@cbracken cbracken changed the title Migrate engine ./tools/gn script (and others) to work with Python 3 Migrate engine tools/gn script (and others) to work with Python 3 May 20, 2021
cbracken added a commit to cbracken/flutter_engine that referenced this issue May 21, 2021
@cbracken
Make tools/gn compatible with Python 3
c4aa488 
Python 2.7 was end-of-lifed on 2020-01-01. This makes tools/gn
compatible with both Python 2.7 and Python 3.

Issue: flutter/flutter#83043
@cbracken cbracken mentioned this issue May 21, 2021
Merged
9 tasks
cbracken added a commit to cbracken/flutter_engine that referenced this issue May 21, 2021
@cbracken
Make tools/gn compatible with Python 3
f3e997a 
Python 2.7 was end-of-lifed on 2020-01-01. This makes tools/gn
compatible with both Python 2.7 and Python 3.

Issue: flutter/flutter#83043
cbracken added a commit to cbracken/flutter_engine that referenced this issue May 21, 2021
@cbracken
Make tools/gn compatible with Python 3
d58931c 
Python 2.7 was end-of-lifed on 2020-01-01. This makes tools/gn
compatible with both Python 2.7 and Python 3.

Issue: flutter/flutter#83043
cbracken added a commit to flutter/engine that referenced this issue May 21, 2021
@cbracken
Make tools/gn compatible with Python 3 (#26321)
27e2ca7 
Python 2.7 was end-of-lifed on 2020-01-01. This makes tools/gn
compatible with both Python 2.7 and Python 3.

Issue: flutter/flutter#83043
@chinmaygarde chinmaygarde added the P3 Issues that are less important to the Flutter project label May 24, 2021
cbracken added a commit to cbracken/buildroot that referenced this issue Jun 4, 2021
@cbracken
Update sysroot_ld_path.py for Python3
dfba56b 
In Python 3, subprocess.check_output returns an encoding-independent
byte array. If passed universal_newlines=True, both Python 2 and 3
decode and return a UTF-8 string.

Issue: flutter/flutter#83043
@cbracken cbracken mentioned this issue Jun 4, 2021
Merged
cbracken added a commit to flutter/buildroot that referenced this issue Jun 4, 2021
@cbracken
Update sysroot_ld_path.py for Python3 (#461)
60e8349 
In Python 3, subprocess.check_output returns an encoding-independent
byte array. If passed universal_newlines=True, both Python 2 and 3
decode and return a UTF-8 string.

Issue: flutter/flutter#83043
cbracken added a commit to cbracken/flutter_engine that referenced this issue Jun 4, 2021
@cbracken
Roll buildroot to 60e8349
1ed68e6 
This picks up:

flutter/buildroot#461: Update sysroot_ld_path.py for Python3

Issue: flutter/flutter#83043
@cbracken cbracken mentioned this issue Jun 4, 2021
Merged
9 tasks
cbracken added a commit to flutter/engine that referenced this issue Jun 5, 2021
@cbracken
Roll buildroot to 60e8349 (#26598)
55065d5 
This picks up:

flutter/buildroot#461: Update sysroot_ld_path.py for Python3

Issue: flutter/flutter#83043
cbracken added a commit to cbracken/buildroot that referenced this issue Jun 5, 2021
@cbracken
Python 3 migration required for Android builds
b720025 
This migrates all Python scripts required to build Android targets to be
Python 3.x compatible.

Main changes:
* Communication with processes now returns bytestreams by default. By
  specifying universal_newlines=True, these streams are decoded as UTF-8.
* md5.update() takes a bytes type, but strings are now utf-8 by default.
  We call string.encode('utf-8') convert to bytes.
* print now uses function syntax

Issue: flutter/flutter#83043
@cbracken cbracken mentioned this issue Jun 5, 2021
Merged
cbracken added a commit to flutter/buildroot that referenced this issue Jun 5, 2021
@cbracken
Python 3 migration required for Android builds (#462)
525709f 
This migrates all Python scripts required to build Android targets to be
Python 3.x compatible.

Main changes:
* Communication with processes now returns bytestreams by default. By
  specifying universal_newlines=True, these streams are decoded as UTF-8.
* md5.update() takes a bytes type, but strings are now utf-8 by default.
  We call string.encode('utf-8') convert to bytes.
* print now uses function syntax

Issue: flutter/flutter#83043
This was referenced Jun 8, 2021
Merged
Merged
Merged
Merged
cbracken added a commit to cbracken/flutter_engine that referenced this issue Jun 9, 2021
@cbracken
Migrate all Python scripts to Python 3
b63f58f 
This migrates all checked in Python scripts to run under Python 3.x.
Python 2.7 reached end-of-life in January of 2020 and has been removed
from may OS distributions at this point.

Rolls buildroot to 607277f

Issue: flutter/flutter#83043
@cbracken cbracken mentioned this issue Jun 9, 2021
Closed
9 tasks
cbracken added a commit to cbracken/flutter_engine that referenced this issue Jun 9, 2021
@cbracken
Migrate all Python scripts to Python 3
e91e5ab 
This migrates all checked in Python scripts to run under Python 3.x.
Python 2.7 reached end-of-life in January of 2020 and has been removed
from may OS distributions at this point.

Rolls buildroot to 607277f

Issue: flutter/flutter#83043
cbracken added a commit to cbracken/flutter_engine that referenced this issue Jun 9, 2021
@cbracken
Migrate all Python scripts to Python 3
5c8947e 
This migrates all checked in Python scripts to run under Python 3.x.
Python 2.7 reached end-of-life in January of 2020 and has been removed
from may OS distributions at this point.

Rolls buildroot to 607277f

Issue: flutter/flutter#83043
cbracken added a commit to cbracken/flutter_engine that referenced this issue Jun 9, 2021
@cbracken
Migrate all Python scripts to Python 3
c04ad6c 
This migrates all checked in Python scripts to run under Python 3.x.
Python 2.7 reached end-of-life in January of 2020 and has been removed
from may OS distributions at this point.

Rolls buildroot to 607277f

Issue: flutter/flutter#83043
cbracken added a commit to cbracken/flutter_engine that referenced this issue Jun 10, 2021
@cbracken
Migrate all Python scripts to Python 3
14f12e8 
This migrates all checked in Python scripts to run under Python 3.x.
Python 2.7 reached end-of-life in January of 2020 and has been removed
from may OS distributions at this point.

Rolls buildroot to 607277f

Issue: flutter/flutter#83043
cbracken added a commit to cbracken/flutter_engine that referenced this issue Jun 10, 2021
@cbracken
Migrate all Python scripts to Python 3
0615e66 
This migrates all checked in Python scripts to run under Python 3.x.
Python 2.7 reached end-of-life in January of 2020 and has been removed
from may OS distributions at this point.

Rolls buildroot to 607277f

Issue: flutter/flutter#83043
@cbracken cbracken mentioned this issue Jul 30, 2021
Merged
cbracken added a commit to cbracken/buildroot that referenced this issue Jul 30, 2021
@cbracken
Replace gyp win_tool.py with tool_wrapper.py
fd48aca 
win_tool.py is incompatible with Python 3. This ports Chromium's
tool_wrapper.py to our buildroot, but doesn't port their full, updated
resource compiler (rc) toolchain.

Chromium moved to a model where they wrap rc in an rc.py script and
support resource compilation on Win, Mac, and Linux. This adds an
additional CIPD dependency that we can avoid. Instead, this ports the
ExecRcWrapper implementation from gyp's win_tool.py, with fixes for
Python 3.

Upstream source:
https://chromium.googlesource.com/chromium/src/+/refs/heads/main/build/toolchain/win

Replaces gyp win_tool.py:
https://chromium.googlesource.com/external/gyp.git/+/refs/heads/master/pylib/gyp/win_tool.py

Issue: flutter/flutter#83043
cbracken added a commit to flutter/buildroot that referenced this issue Jul 30, 2021
@cbracken
Replace gyp win_tool.py with tool_wrapper.py (#492)
ed767ed 
win_tool.py is incompatible with Python 3. This ports Chromium's
tool_wrapper.py to our buildroot, but doesn't port their full, updated
resource compiler (rc) toolchain.

Chromium moved to a model where they wrap rc in an rc.py script and
support resource compilation on Win, Mac, and Linux. This adds an
additional CIPD dependency that we can avoid. Instead, this ports the
ExecRcWrapper implementation from gyp's win_tool.py, with fixes for
Python 3.

Upstream source:
https://chromium.googlesource.com/chromium/src/+/refs/heads/main/build/toolchain/win

Replaces gyp win_tool.py:
https://chromium.googlesource.com/external/gyp.git/+/refs/heads/master/pylib/gyp/win_tool.py

Issue: flutter/flutter#83043
This was referenced Jul 30, 2021
Merged
Merged
cbracken added a commit to cbracken/buildroot that referenced this issue Jul 30, 2021
@cbracken
Use Python 3 for all gn exec_script calls
270963b 
This switches the default script runtime from python/python.exe to
python3/python3.exe for all exec_script calls invoked from gn build
files.

Issue: flutter/flutter#83043
@cbracken cbracken mentioned this issue Jul 30, 2021
Merged
cbracken added a commit to flutter/buildroot that referenced this issue Jul 31, 2021
@cbracken
Use Python 3 for all gn exec_script calls (#494)
630be43 
This switches the default script runtime from python/python.exe to
python3/python3.exe for all exec_script calls invoked from gn build
files.

Issue: flutter/flutter#83043
cbracken added a commit to cbracken/flutter_engine that referenced this issue Jul 31, 2021
@cbracken
Enable Python 3 for all gn exec_script calls
f847028 
This rolls buildroot to 630be43.

Fixes: flutter/flutter#83043
@cbracken cbracken mentioned this issue Jul 31, 2021
Merged
8 tasks
cbracken added a commit to cbracken/flutter_engine that referenced this issue Jul 31, 2021
@cbracken
Remove dependency on gyp
923624b 
This removes our dependency on gyp. We migrated from gyp to gn more than
5 years ago. The only remaining dependency was on the win_tool.py script
in our Windows toolchain rules. That dependency was removed in:
flutter/buildroot@ed767ed

Bug: flutter/flutter#83043
@cbracken cbracken mentioned this issue Jul 31, 2021
Closed
8 tasks
@cbracken cbracken mentioned this issue Jul 31, 2021
Merged
8 tasks
cbracken added a commit to cbracken/flutter_engine that referenced this issue Jul 31, 2021
@cbracken
Migrate all Python hashbangs to Python 3
4f2e9dc 
Migrates all `#!/usr/bin/env python` hashbang lines to use python3.

Issue: flutter/flutter#83043
cbracken added a commit to cbracken/flutter_engine that referenced this issue Jul 31, 2021
@cbracken
Migrate all Python hashbangs to Python 3
5ff134b 
Migrates all `#!/usr/bin/env python` hashbang lines to use python3.

Also updates the licence tool to treat python3 scripts on par with
python 2 scripts.

Issue: flutter/flutter#83043
@cbracken cbracken reopened this Jul 31, 2021
@cbracken
Copy link
Member Author

@safasofuoglu thanks for the update and sorry this slipped off my radar while I focused on some Google internal work. A couple more patches and this migration should be complete.

cbracken added a commit to flutter/engine that referenced this issue Jul 31, 2021
@cbracken
Migrate all Python hashbangs to Python 3 (#27838)
9985bc0 
Migrates all `#!/usr/bin/env python` hashbang lines to use python3.

Also updates the licence tool to treat python3 scripts on par with
python 2 scripts.

Issue: flutter/flutter#83043
@cbracken
Copy link
Member Author

cbracken commented Aug 2, 2021

As of flutter/engine#27845, all python scripts in the flutter/engine and flutter/buildroot repos are migrated to Python 3.

cbracken added a commit to cbracken/flutter that referenced this issue Aug 5, 2021
@cbracken
Migrate python invocations to python3
c3a1703 
Python 2 has been end-of-life'd since January 2020, all Python scripts
on which Flutter depends have been migrated to Python 3. This changes
documentation and remaining invocations to explicitly invoke python3.

Issue: flutter#83043
@cbracken cbracken mentioned this issue Aug 5, 2021
Merged
8 tasks
cbracken added a commit that referenced this issue Aug 6, 2021
@cbracken
Migrate python invocations to python3 (#87759)
497eb13 
Python 2 has been end-of-life'd since January 2020, all Python scripts
on which Flutter depends have been migrated to Python 3. This changes
documentation and remaining invocations to explicitly invoke python3.

Issue: #83043
@cbracken
Copy link
Member Author

cbracken commented Aug 6, 2021

This is also complete for flutter/flutter as of #87759.

@github-actions
Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 20, 2021
naudzghebre pushed a commit to naudzghebre/engine that referenced this issue Sep 2, 2021
@cbracken
Enable Python 3 for all gn exec_script calls (flutter#27835)
7ac6d7a 
This rolls buildroot to 630be43.

Fixes: flutter/flutter#83043
filmil pushed a commit to filmil/engine that referenced this issue Apr 21, 2022
@cbracken @filmil
Enable Python 3 for all gn exec_script calls (flutter#27835)
a4038e0 
This rolls buildroot to 630be43.

Fixes: flutter/flutter#83043
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@cbracken cbracken

Labels
engine flutter/engine repository. See also e: labels. P3 Issues that are less important to the Flutter project team Infra upgrades, team productivity, code health, technical debt. See also team: labels.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enable Python 3 for all gn exec_script calls cbracken/flutter_engine
3 participants
@chinmaygarde @cbracken @safasofuoglu