Upgrade NVDA codebase to Python 3

[josephsl]

Hi,

This issue serves as an umbrella issue for several related issues, one of them being #7077.

Python 3 introduces changes to syntax and inner workings of Python. Some of the changes include:

• Unicode strings.
• Use range function.
• _winreg -> winreg.
• threading.Barrier.
• collections.ChainMap.
• Reorganized standard library folder structure.
• Timer-based GIL instead of instruction count.
• Possibilities with asynchronous functions (async/await).
• Virtual environments.
• Built-in compatibility with Windows 10 (Python 3.5 or later).
• Losing Windows XP support (Python 3.5 and later).
• No more .pyo files.

Upgrading to Python 3 will require huge coordination, as NVDA developers, add-on writers, translators, users and others will need to adapt new attitudes, assumptions and so on. In terms of dependencies, the only thing left is SCons (wxPython is discussed in #7077).

Thanks.

[OzancanKaratas]

Python's 2to3 script can be used for this request. Its developers included 2to3 into Python 3.

[josephsl]

Hi,

A few years ago, I tried this approach and it produced mixed results. Although 2to3 did work to some extent, the resulting code needed corrections. Also, this is just not as easy as letting the current code run through this script: dependencies must be satisfied, and we need just two more dependencies in order to even start transition planning, namely SCons and wxPython (wxPython is another story, which is covered in #7077).

Thanks.

[josephsl]

Hi,

Although the actual transition plan is a bit cloudy at the moment, @jcsteh suggests via IRC that next year would be a good timeframe, and @LeonarddeR concurring with this thinking. A tentative plan is thus:

Prerequisites: wxPython 4 testing must be completed i.e. merged into master and tested in at least two NVDA stable releases. For the purposes of this plan, N refers to NVDA release with wxPython 4, N+1 referring to next one and so on.

1. Proposal and research: a suitable Python 3 release must be chosen and should be aimed for during the transition. At the moment Python 3.5 is the lowest practical version due to availability of wxPython 4 wheels for it. Ideally, for maximum market availability, Python 3.4 should be targeted as it is the last version to support Windows XP. Based on the proposed time frame below, the most practical version to aim for is 3.6, with 3.7 coming out in 2018.
2. Issue identification: once a suitable Python 3 release is chosen, showstopper issues must be identified. These include incompatible syntax, deprecated code paths and modules, replacement modules and so on. This cannot commence until NVDA N is released.
3. Preparation: developers (Core and add-ons) should learn about differences between Python 2.7 and the chosen 3.x release while NVDA N+1 and N+2 are under development. This is also the ideal time frame to research fixes and workarounds for major issues identified.
4. Transition workflow: once issues were identified and all the dependencies are satisfied, transition work can begin. The ideal timeframe is NVDAN+3 and N+4 development. During this time transition work will be incubated in next.
5. Add-ons: at the start of NVDA N+4 development, add-ons community will be asked to start porting their code to Python 3. A few authors will do some work during N+3 development. Most of this should be done no later than next release after Python 3 work is integrated into master branch. For example, if Python 3 work is done by NVDA 2018.4 master, the work from add-ons community should be done by 2019.1 (or 2019.2 at the latest).
6. Aftermath: any dependencies on Python 2.7 should be kept to a minimum, or if to be removed, should be done when it is ready.

The above plan is subject to change. Thanks.

[nishimotz]

Is it possible to build a NVDA distribution, which contains selectable Python 2.7 runtime and Python 3.x runtime simultaneously?
If single version of NVDA can be switchable between Python 2.7 mode and Python 3.x mode during the migration period (for several official releases), possible benefits are:

• the official NVDA build will not be separated during the migration. In other words, I my concern is the separation of two official versions of NVDA binary, which are based on Python 2.7 and 3.x.
• the evaluators or add-on developers of NVDA can compare the behavior of apps or add-ons easily without using two distributions of NVDA.
• the user can easily choose between sticking around 2.7-based NVDA or testing 3.x-based NVDA during the migration period.
• the release management will be flexible, because the real change is just to switch the default mode from 2.7-based mode to 3.x-based mode. During the migration period, the usage statistics can be obtained to judge that 3.x-based mode is mature or not.

[jcsteh]

Thanks for your thoughts. I don't think having both versions in the same distribution is feasible/workable:

• There is probably at least some code that cannot be compatible with both Python 2 and 3; it must be one or the other. This means the Python 3 work has to be kept in a separate branch.
• I think this whole concept is too complicated for many users to grasp. It's reasonable to expect that those who have some understanding of such issues (and are willing to test) will be quite able to separately test another branch.
• We'd effectively be shipping code in a final release that we aren't certain is ready for use in production. That goes against the idea of a final release.
• All the Python byte code and extension dlls would be different between the two versions.
• We wouldn't be able to have both versions in the same directory due to file naming conflicts.
• It would significantly increase the size of the download (probably near double).
• Our build system would require major refactoring to allow for this, if it's even possible.
• A user would still have to select which executable to run anyway, so they're pretty much separate except for the download.

[Brian1Gaff]

Would you maybe consider that by the time Python 3 versions are stable that it be only a 64 bit app? Or can it be made to install in both 64 and 32 bit systems. It would get rid of the issue of people attempting to install it in xp, as will I am sure happen otherwise. Brian

[LeonarddeR]

Ideally, for maximum market availability, Python 3.4 should be targeted as it is the last version to support Windows XP.

I don't want to start the whole Xp discussion over again (#6718), but I disagree with this. When moving to Python 3, we should target the last version available at the time of the switch.

[josephsl]

Hi, hence the “ideal” case. In reality, 3.5 or higher should be targeted (3.6 is optimal), hence we can officially start working on #6718 as well. Thanks.

[LeonarddeR]

Thanks for your clarification.

Just for reference, one major argument for not targetting 3.4 is this overview. Note that the EOL date for Python 3.4 is 2019-03-16. After this date, major security risks won't be patched any more.

[josephsl]

Hi,

Laying the initial foundation in an internal branch (located at a personal fork of NVDA source code):

The trivial case at this time is winreg versus _winreg compatibility. Because API's and side effects are the same, this is perhaps the easiest milestone: import winreg, and if not, use _winreg and rename it to winreg. I'm intentionally using the newer winreg.* methods to incorporate forward thinking principles.

For future reference: if anyone wants to investigate Python 3 compatibility (for personal research purposes), or wish to test and send PR's, please do NOT send them to NV Access yet (there are higher priority works to be done, hence I propose p3 for this year). Instead, use josephsl/NVDA branch py3000 for now.

In doing this, I recommend folks to follow the below advice and guidelines:

1. Become familiar with python 3 syntax, some inner workings, and standard library folder structure.
2. Install Python 3.5 or later (Python 3.4 won't cut it, as noted above).
3. Test a concept in Python 3 interpreter first before applying to NVDA.
4. Use 2to3 script with -f all (or a combination of needed fixers) with a diff output being sent to a text file for examination later. This should be done regularly.
5. After modifying each file, run NVDA to make sure your edits didn't break features or introduce regressions.
6. For now, try using compatibility approach so NVDA can at least run on 2.7 and 3.5/3.6/3.x. This should be done until SCons and/or wxPython and what not are ready.
7. If you are aiming for stricter compliance, use Python 3 syntax and modules from the beginning.
8. Get Python experts involved so they can serve as additional reviewers and help us iron out things here and there before the official transition begins.

Note: the above procedure is for those who're thinking about tackling this issue for personal research purposes. During this process, someone should write a report to be sent to NV Access, collecting our data and findings.

When meeting ambiguous code or code paths that could work well on both 2 and 3: try either strict compliance or compatibility approach. The following winreg and dictionary key iteration gives examples of these approaches:

• Imports:

Strict: import winreg and use winreg.*
Compatibility: import _winreg as winreg and use winreg.*, or use import winreg as _winreg and use existing code paths without modifications.

• Dictionary key iteration:

Strict: for key in dict.keys(): pass
Compatibility: Have a flag that checks sys.version.startswith("3") and: for key in dict.keys() if py3 else dict.iterkeys(): pass

Once you choose an approach, please do not be tempted to switch methods, otherwise you'll end up with more headaches than before.

Thanks.

[ctoth]

I am concerned about the long tail of addons that will either never be ported or will require community intervention or otherwise will be slow to Python 3 and recommend moving up the timeline for pushing addon developers into at least using the subset of Python that is compatible with both versions to as early as possible.
I further recommend including something like Python Future in core to ease the porting process.

[josephsl]

To @ctoth: As for add-ons, I'm taking your advice by making sure future versions of my add-ons are source-compatible with both 2 and 3. Sometime this year, I'll advise others to follow suit. Thanks.

[josephsl]

Follow-up to SCons: see #7520 for an update. With that in place, we're ready to start researching. Thanks.

[josephsl]

Waiting for #7546 to come to master branch before continuing. Thanks.

[josephsl]

As noted on #7520, SCons 3 is out, and #7546 is part of master, so let's start researching. Thanks.

[josephsl]

Update: Once #7568 is part of NVDA master, that's when I propose code should be checked in. Thanks.

[LeonarddeR]

I belief that first, all dependencies for Python 3 should be satisfied, so this includes the update to WX Python Phoenix 4.0 being incorporated into Master.

[josephsl]

Hi, yep, that’ll be done as part of a pull request once wxPython 4 officially ships (now up to beta 2). Thanks.

[Brian1Gaff]

If I may make a users comment here, using my idiot head.."
There sould, in my opinion be a new snapshot branch that we can test. Also when thinking about add ons, maybe some kind of protection should be put into the loading of add ons so nobody can load the old ones, only the new ones in the new code versions of nvda.

As you will want testing to at least be done by as many basic users as possible when the major issues are corrected, it is then when it can be put out in a master.
Also, because of changes in syntax, does this mean app modules already in nvda will possibly need to be rewritten? If yes, could I ask that we still support legacy software aslong as people have it running on at least windows 7. That will stop a lot of problems trying to solve things later on.
I and others do use such software, but I'm not sure how many of these use specific app modules or just work m in nvda anyway. Outlook Express/live mail is one, andprobably older versions of Office like 2007, and several third party apps as well.

As I've said before, I'm OK to test at present with portable versions of new builds in my bog standard windows 7. I'm expecting to get some more modern machines soon so will be up to speed then, but do have a history of using legacy software simply cos it works.

Lastly in this issue, there are comments about xp, but as you have already dropped support there, don't let it mean that you also drop support for legacy software as well.
I think blind people on average do hang on to older software longer than most simply to avoid the learning curve. I am now 67, and find my learning curve is somewhat harder to climb than when I was 25!

[josephsl]

Hi, I’ll make sure all app modules that ships with NVDA are properly ported, or if not completely, at least made compatible with both 2 and 3. Thanks.

[LeonarddeR]

We are considering adding the -3 argument to our python 2 build, this should highlight problems in master which can be made compatible with both 2 and 3. This will reduce the burden of merges, however it is likely to push some risk back into prior releases.

Hmm, this seems like an interesting idea, though I wonder whether py2exe can deal wit this while freezing the code. That needs to be found out.
It is probably very trivial to add the -3 flag to test builds only, so release builds don't get affected.

[feerrenrut]

I'm going to close this issue now, as I think it has served it's purpose. It has been superceded by the Python 3 project board