Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade NVDA codebase to Python 3 #7105

Closed
josephsl opened this issue Apr 26, 2017 · 38 comments
Closed

Upgrade NVDA codebase to Python 3 #7105

josephsl opened this issue Apr 26, 2017 · 38 comments
Assignees
Labels
help wanted Python 3

Comments

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Apr 26, 2017

Hi,

This issue serves as an umbrella issue for several related issues, one of them being #7077.

Python 3 introduces changes to syntax and inner workings of Python. Some of the changes include:

  • Unicode strings.
  • Use range function.
  • _winreg -> winreg.
  • threading.Barrier.
  • collections.ChainMap.
  • Reorganized standard library folder structure.
  • Timer-based GIL instead of instruction count.
  • Possibilities with asynchronous functions (async/await).
  • Virtual environments.
  • Built-in compatibility with Windows 10 (Python 3.5 or later).
  • Losing Windows XP support (Python 3.5 and later).
  • No more .pyo files.

Upgrading to Python 3 will require huge coordination, as NVDA developers, add-on writers, translators, users and others will need to adapt new attitudes, assumptions and so on. In terms of dependencies, the only thing left is SCons (wxPython is discussed in #7077).

Thanks.

@OzancanKaratas
Copy link
Contributor

@OzancanKaratas OzancanKaratas commented Apr 26, 2017

Python's 2to3 script can be used for this request. Its developers included 2to3 into Python 3.

@josephsl
Copy link
Collaborator Author

@josephsl josephsl commented Apr 26, 2017

Hi,

A few years ago, I tried this approach and it produced mixed results. Although 2to3 did work to some extent, the resulting code needed corrections. Also, this is just not as easy as letting the current code run through this script: dependencies must be satisfied, and we need just two more dependencies in order to even start transition planning, namely SCons and wxPython (wxPython is another story, which is covered in #7077).

Thanks.

@josephsl
Copy link
Collaborator Author

@josephsl josephsl commented Apr 26, 2017

Hi,

Although the actual transition plan is a bit cloudy at the moment, @jcsteh suggests via IRC that next year would be a good timeframe, and @leonardder concurring with this thinking. A tentative plan is thus:

Prerequisites: wxPython 4 testing must be completed i.e. merged into master and tested in at least two NVDA stable releases. For the purposes of this plan, N refers to NVDA release with wxPython 4, N+1 referring to next one and so on.

  1. Proposal and research: a suitable Python 3 release must be chosen and should be aimed for during the transition. At the moment Python 3.5 is the lowest practical version due to availability of wxPython 4 wheels for it. Ideally, for maximum market availability, Python 3.4 should be targeted as it is the last version to support Windows XP. Based on the proposed time frame below, the most practical version to aim for is 3.6, with 3.7 coming out in 2018.
  2. Issue identification: once a suitable Python 3 release is chosen, showstopper issues must be identified. These include incompatible syntax, deprecated code paths and modules, replacement modules and so on. This cannot commence until NVDA N is released.
  3. Preparation: developers (Core and add-ons) should learn about differences between Python 2.7 and the chosen 3.x release while NVDA N+1 and N+2 are under development. This is also the ideal time frame to research fixes and workarounds for major issues identified.
  4. Transition workflow: once issues were identified and all the dependencies are satisfied, transition work can begin. The ideal timeframe is NVDAN+3 and N+4 development. During this time transition work will be incubated in next.
  5. Add-ons: at the start of NVDA N+4 development, add-ons community will be asked to start porting their code to Python 3. A few authors will do some work during N+3 development. Most of this should be done no later than next release after Python 3 work is integrated into master branch. For example, if Python 3 work is done by NVDA 2018.4 master, the work from add-ons community should be done by 2019.1 (or 2019.2 at the latest).
  6. Aftermath: any dependencies on Python 2.7 should be kept to a minimum, or if to be removed, should be done when it is ready.

The above plan is subject to change. Thanks.

@nishimotz
Copy link
Contributor

@nishimotz nishimotz commented Apr 27, 2017

Is it possible to build a NVDA distribution, which contains selectable Python 2.7 runtime and Python 3.x runtime simultaneously?
If single version of NVDA can be switchable between Python 2.7 mode and Python 3.x mode during the migration period (for several official releases), possible benefits are:

  • the official NVDA build will not be separated during the migration. In other words, I my concern is the separation of two official versions of NVDA binary, which are based on Python 2.7 and 3.x.
  • the evaluators or add-on developers of NVDA can compare the behavior of apps or add-ons easily without using two distributions of NVDA.
  • the user can easily choose between sticking around 2.7-based NVDA or testing 3.x-based NVDA during the migration period.
  • the release management will be flexible, because the real change is just to switch the default mode from 2.7-based mode to 3.x-based mode. During the migration period, the usage statistics can be obtained to judge that 3.x-based mode is mature or not.

@jcsteh
Copy link
Contributor

@jcsteh jcsteh commented Apr 27, 2017

Thanks for your thoughts. I don't think having both versions in the same distribution is feasible/workable:

  • There is probably at least some code that cannot be compatible with both Python 2 and 3; it must be one or the other. This means the Python 3 work has to be kept in a separate branch.
  • I think this whole concept is too complicated for many users to grasp. It's reasonable to expect that those who have some understanding of such issues (and are willing to test) will be quite able to separately test another branch.
  • We'd effectively be shipping code in a final release that we aren't certain is ready for use in production. That goes against the idea of a final release.
  • All the Python byte code and extension dlls would be different between the two versions.
  • We wouldn't be able to have both versions in the same directory due to file naming conflicts.
  • It would significantly increase the size of the download (probably near double).
  • Our build system would require major refactoring to allow for this, if it's even possible.
  • A user would still have to select which executable to run anyway, so they're pretty much separate except for the download.

@Brian1Gaff
Copy link

@Brian1Gaff Brian1Gaff commented Apr 27, 2017

@leonardder
Copy link
Collaborator

@leonardder leonardder commented Apr 27, 2017

Ideally, for maximum market availability, Python 3.4 should be targeted as it is the last version to support Windows XP.

I don't want to start the whole Xp discussion over again (#6718), but I disagree with this. When moving to Python 3, we should target the last version available at the time of the switch.

@josephsl
Copy link
Collaborator Author

@josephsl josephsl commented Apr 27, 2017

@leonardder
Copy link
Collaborator

@leonardder leonardder commented Apr 27, 2017

Thanks for your clarification.

Just for reference, one major argument for not targetting 3.4 is this overview. Note that the EOL date for Python 3.4 is 2019-03-16. After this date, major security risks won't be patched any more.

@josephsl
Copy link
Collaborator Author

@josephsl josephsl commented Jul 18, 2017

Hi,

Laying the initial foundation in an internal branch (located at a personal fork of NVDA source code):

The trivial case at this time is winreg versus _winreg compatibility. Because API's and side effects are the same, this is perhaps the easiest milestone: import winreg, and if not, use _winreg and rename it to winreg. I'm intentionally using the newer winreg.* methods to incorporate forward thinking principles.

For future reference: if anyone wants to investigate Python 3 compatibility (for personal research purposes), or wish to test and send PR's, please do NOT send them to NV Access yet (there are higher priority works to be done, hence I propose p3 for this year). Instead, use josephsl/NVDA branch py3000 for now.

In doing this, I recommend folks to follow the below advice and guidelines:

  1. Become familiar with python 3 syntax, some inner workings, and standard library folder structure.
  2. Install Python 3.5 or later (Python 3.4 won't cut it, as noted above).
  3. Test a concept in Python 3 interpreter first before applying to NVDA.
  4. Use 2to3 script with -f all (or a combination of needed fixers) with a diff output being sent to a text file for examination later. This should be done regularly.
  5. After modifying each file, run NVDA to make sure your edits didn't break features or introduce regressions.
  6. For now, try using compatibility approach so NVDA can at least run on 2.7 and 3.5/3.6/3.x. This should be done until SCons and/or wxPython and what not are ready.
  7. If you are aiming for stricter compliance, use Python 3 syntax and modules from the beginning.
  8. Get Python experts involved so they can serve as additional reviewers and help us iron out things here and there before the official transition begins.

Note: the above procedure is for those who're thinking about tackling this issue for personal research purposes. During this process, someone should write a report to be sent to NV Access, collecting our data and findings.

When meeting ambiguous code or code paths that could work well on both 2 and 3: try either strict compliance or compatibility approach. The following winreg and dictionary key iteration gives examples of these approaches:

  • Imports:

Strict: import winreg and use winreg.*
Compatibility: import _winreg as winreg and use winreg.*, or use import winreg as _winreg and use existing code paths without modifications.

  • Dictionary key iteration:

Strict: for key in dict.keys(): pass
Compatibility: Have a flag that checks sys.version.startswith("3") and: for key in dict.keys() if py3 else dict.iterkeys(): pass

Once you choose an approach, please do not be tempted to switch methods, otherwise you'll end up with more headaches than before.

Thanks.

@ctoth
Copy link
Contributor

@ctoth ctoth commented Jul 22, 2017

I am concerned about the long tail of addons that will either never be ported or will require community intervention or otherwise will be slow to Python 3 and recommend moving up the timeline for pushing addon developers into at least using the subset of Python that is compatible with both versions to as early as possible.
I further recommend including something like Python Future in core to ease the porting process.

@josephsl
Copy link
Collaborator Author

@josephsl josephsl commented Aug 25, 2017

To @ctoth: As for add-ons, I'm taking your advice by making sure future versions of my add-ons are source-compatible with both 2 and 3. Sometime this year, I'll advise others to follow suit. Thanks.

@josephsl
Copy link
Collaborator Author

@josephsl josephsl commented Aug 25, 2017

Follow-up to SCons: see #7520 for an update. With that in place, we're ready to start researching. Thanks.

@josephsl
Copy link
Collaborator Author

@josephsl josephsl commented Aug 31, 2017

Waiting for #7546 to come to master branch before continuing. Thanks.

josephsl added a commit to josephsl/nvda that referenced this issue Sep 19, 2017
Before we can move to Python 3 (nvaccess#7105), two important dependencies must be satisfied: wxPython 4 (nvaccess#7077) and SCons (nvaccess#7520). As of September 18, 2017, both were satisfied, although wxPython 4 is still in beta testing phase.
For now, upgrade SCons to 3.0.0, which supports Python 2.7, 3.5, and 3.6. Key changes include changing print statements to print function and no more Environment.gettextMoFile attribute unless it has changed somehow. Upgrading to SCons 3 is one of the prerequisites to transitioning to Python 3.x.
@josephsl
Copy link
Collaborator Author

@josephsl josephsl commented Sep 19, 2017

As noted on #7520, SCons 3 is out, and #7546 is part of master, so let's start researching. Thanks.

@josephsl
Copy link
Collaborator Author

@josephsl josephsl commented Sep 19, 2017

Update: Once #7568 is part of NVDA master, that's when I propose code should be checked in. Thanks.

@leonardder
Copy link
Collaborator

@leonardder leonardder commented Sep 20, 2017

I belief that first, all dependencies for Python 3 should be satisfied, so this includes the update to WX Python Phoenix 4.0 being incorporated into Master.

@josephsl
Copy link
Collaborator Author

@josephsl josephsl commented Sep 20, 2017

@Brian1Gaff
Copy link

@Brian1Gaff Brian1Gaff commented Sep 24, 2017

If I may make a users comment here, using my idiot head.."
There sould, in my opinion be a new snapshot branch that we can test. Also when thinking about add ons, maybe some kind of protection should be put into the loading of add ons so nobody can load the old ones, only the new ones in the new code versions of nvda.

As you will want testing to at least be done by as many basic users as possible when the major issues are corrected, it is then when it can be put out in a master.
Also, because of changes in syntax, does this mean app modules already in nvda will possibly need to be rewritten? If yes, could I ask that we still support legacy software aslong as people have it running on at least windows 7. That will stop a lot of problems trying to solve things later on.
I and others do use such software, but I'm not sure how many of these use specific app modules or just work m in nvda anyway. Outlook Express/live mail is one, andprobably older versions of Office like 2007, and several third party apps as well.

As I've said before, I'm OK to test at present with portable versions of new builds in my bog standard windows 7. I'm expecting to get some more modern machines soon so will be up to speed then, but do have a history of using legacy software simply cos it works.

Lastly in this issue, there are comments about xp, but as you have already dropped support there, don't let it mean that you also drop support for legacy software as well.
I think blind people on average do hang on to older software longer than most simply to avoid the learning curve. I am now 67, and find my learning curve is somewhat harder to climb than when I was 25!

@josephsl
Copy link
Collaborator Author

@josephsl josephsl commented Sep 24, 2017

@leonardder
Copy link
Collaborator

@leonardder leonardder commented May 21, 2019

We are considering adding the -3 argument to our python 2 build, this should highlight problems in master which can be made compatible with both 2 and 3. This will reduce the burden of merges, however it is likely to push some risk back into prior releases.

Hmm, this seems like an interesting idea, though I wonder whether py2exe can deal wit this while freezing the code. That needs to be found out.
It is probably very trivial to add the -3 flag to test builds only, so release builds don't get affected.

josephsl added a commit to josephsl/nvda that referenced this issue May 23, 2019
Because various calls to xrange are used by generators, change them by calling range function (in Python 3, range returns generators).
josephsl added a commit to josephsl/nvda that referenced this issue May 24, 2019
josephsl added a commit to josephsl/nvda that referenced this issue May 24, 2019
…cess#7105.

Comment from Reef Turner (NV Access): add explanatory comment, prefixing the comment with '# Py3 review required'. Because some changes are based on another issue (nvaccess#9067, for example), prefix the review comment with issue number references as appropriate.
josephsl added a commit to josephsl/nvda that referenced this issue May 25, 2019
josephsl added a commit to josephsl/nvda that referenced this issue May 25, 2019
josephsl added a commit to josephsl/nvda that referenced this issue May 25, 2019
… statement changes in Python 3, applicable to log viewer and input core. Re nvaccess#7105.
josephsl added a commit to josephsl/nvda that referenced this issue May 25, 2019
josephsl added a commit to josephsl/nvda that referenced this issue May 26, 2019
…tually no longer needed in Python 3. Re nvaccess#7105.

In order to deal with Unicode and non-Unicode paths, a dedicated function was provided to convert a package path to Unicode and return it. Since Python 3 can handle Unicode directly, consider removing this function altogether if no longer needed.
@feerrenrut feerrenrut self-assigned this Jun 4, 2019
@feerrenrut
Copy link
Member

@feerrenrut feerrenrut commented Jun 28, 2019

I'm going to close this issue now, as I think it has served it's purpose. It has been superceded by the Python 3 project board

Update NVDA to Python 3 automation moved this from To do to Done Jun 28, 2019
josephsl added a commit to josephsl/nvda that referenced this issue Jul 3, 2019
josephsl added a commit to josephsl/nvda that referenced this issue Jul 17, 2019
josephsl added a commit to josephsl/nvda that referenced this issue Jul 17, 2019
josephsl added a commit to josephsl/nvda that referenced this issue Jul 27, 2019
JulienCochuyt added a commit to accessolutions/nvda that referenced this issue Sep 20, 2019
JulienCochuyt added a commit to accessolutions/nvda that referenced this issue Sep 20, 2019
…#7105)

Impacts both the add-ons state and the updater state files.
Both are now written using pickle protocol 0 so that Python 2 version of NVDA
can read them in case of downgrading.

Re nvaccess#10224 (comment)
michaelDCurran pushed a commit that referenced this issue Sep 23, 2019
…10255)

Impacts both the add-ons state and the updater state files.
Both are now written using pickle protocol 0 so that Python 2 version of NVDA
can read them in case of downgrading.

Re #10224 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Python 3
Projects
No open projects
Development

No branches or pull requests

10 participants