Merge Python 3 staging to threshold #9929
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* App module handler/Python 3: print statement -> function when printing crash dump path. * SAPI5 synth driver/Python 3: print statement -> function when announcing function hooks. * SAPI 5: print function -> log.debug. Reviewed and clarified by Mick Curran (NV Access): print function should have been a log.debug call.
* Add-on handler and update check/Python 3: file function -> with open function for reading from and writing to pickle files. Re #9038. * NVDA helper and watchdog/Python 3: with file -> with open function. Re #9038. * Config/Python 3: use open function when creating an empty profile data file. Re #9038. * Build version/Python 3: use with open function when obtaining Git head/commit for version information construction. Re #9038. Without modifying head/commit values for version info, NVDA will report 'year.major.minor' when running from source. Therefore edit this routine. * Log viewer/Python 3: add an explanatory note on with open function and encoding issue. Re #9038. When saving logs, a log file object will be created. The routine responsible for this is eligible for conversion to 'with open' statement, but there is a note about UTF-8 codecs conversion problem. Thus add a note regarding this and exclude that routine from conversion for now. * Add-on handler and update check/Python 3: use binary format when working with pickles. Re #9038. Clarified by Mick Curran (NV Access): in Python 3, when loading and dumping pickles, binary format must be used (rb/wb). * Log viewer/Python 3: save log file in UTF-8 format. Re #9038. Reviewed by Mick Curran (NV Access): open log file for saving with UTF-8 encoding from the start. Note that Python 2's open function does not include encoding parameter directly, but Python 3 does.
…sion operations (#9692) * GUI/Python 3: use floor division (//) when obtaining pixel coordinates in integers. Re #9641. Python 3 changes one slash to tru division rather than classic division. Because of this, when trying to obtain pixel coordinates, float is returned. Thus use floor division (//) to guarantee integers. * NVWare/Python 3: use floor division for int values. Re #9641. ctypes.wintypes.WORD is mapped to ctypes.USHORT, which is in fact an integer. Thus use two slashes instead of one in order to guarantee this. * OneCore speech synthesizer/Python 3: ensure integers via floor division. Re #9641. A comment says no floating points. One way to ensure this is use of two slashes in Python 3.
…on 3 (#9671) * Various modules/Python 3: wrap dict.items/keys/values inside a list call. Re #9067. Modules such as UIA handler, settings dialogs and others use dict.items/keys/values. This means in Python 2, it returns a list, whereas it returns an iterator in Python 3. Therefore wrap these inside a list call to preserve semantics (also include notes for some of these). * NVDAObjects/Python 3: dict.iteritems -> dict.items. Re #9067. * IAccessible handler/Python 3: dict.iteritems -> dict.items. Re #9067. * UIA handler and utilities/Python 3: dict.iteritems -> dict.items, dict.iterkeys -> dict.keys. Re #9067. * Add-on handler/Python 3: dict.itervalues -> dict.values. Re #9067. * App module handler/Python 3: dict.iteritems -> dict.items, dict.itervalues -> dict.values. Re #9067. * Base object/Python 3: dict.iteritems -> dict.items. Re #9067. * Braille display detection/Python 3: dict.iteritems -> dict.items. Re #9067 * Braille display drivers/Python 3: dict.iteritems -> dict.items, dict.itervalues -> dict.values. Re #9067. * Character processing/Python 3: dict.iteritems -> dict.items, dict.itervalues -> dict.values. Re #9067. * Config/Python 3: dict.iteritems -> dict.items. Re #9067. * Global plugin handler/Python 3: dict.iteritems -> dict.items. Re #9067. * GUI/Python 3: dict.iteritems -> dict.items, dict.itervalues -> dict.values. Re #9067. * Input core/Python 3: dict.iteritems -> dict.items. Re #9067. * Installer/Python 3: dict.iteritems -> dict.items. Re #9067. * Oleacc/Python 3: dict.iteritems -> dict.items. Re #9067. * Speech/Python 3: dict.iteritems -> dict.items. Re #9067. * Speech XML/Python 3: dict.iteritems -> dict.items. Re #9067. * Synth drivers/Python 3: dict.iteritems -> dict.items, dict.itervalues -> dict.values. Re #9067. * Table utils/Python 3: dict.iteritems -> dict.items. Re #9067. * Text infos offsets/Python 3: dict.iteritems -> dict.items. Re #9067. * Touch tracker/Python 3: dict.itervalues -> dict.values. Re #9067. * VK codes/Python 3: dict.iteritems -> dict.items. Re #9067. * Watchdog/Python 3: dict.iteritems -> dict.items. Re #9067.
* Text infos/offsets/Python 3: remove six.moves.range import. Re #9078. Although a good compatibility call, it is better to call range function directly. The only place six.moves.range was used was textInfos/offsets. * Python 3/API handlers: xrange -> range, explanatory comments. Re #9078. Xrange is gone in Python 3, replaced by range which performs the same thing as xrange. The following API handlers were modified: IAccessible, JAB, UIA. * NVDA objects/top level/Python 3: xrange -> range and explanatory comments. Re #9078. * NVDAObjects/IAccessible/Python 3: xrange -> range and explanatory comments. Re #9078. * NVDAObjects/JAB/Python 3: xrange -> range and explanatory comments. Re #9078. * NVDAObjects/UIA/Python 3: xrange -> range and explanatory comments. Re #9078. * NVDAObjects/window/Python 3: xrange -> range and explanatory comments. Re #9078. * NVDA helper/Python 3: xrange -> range. Re #9078. * UIA browse mode and utilities: xrange -> range. Re #9078. * API/Python 3: xrange -> range for states set construction and focus ancestor calcualtions. Re #9078. * App modules/Python 3: xrange -> range except for Skype module. Re #9078. If Skype app module is planned to be removed, it won't make sense to change xrnage to range. Other than this, other app modules were edited, including Excel, Kindle, Miranda, Outlook, and VIPMud. * Braille input and output/Python 3: xrange -> range. Re #9078. * Braille display drivers/Python 3: xrange -> range, mostly involving device identification, status, input devices and connection attempts. Re #9078. * Config/Python 3: xrange -> range. Re #9078. * Display model/Python 3: xrange -> range. Re #9078. * GUI/NVDA controls/Python 3: xrange -> range, mostly dealing with custom check list box handling. Re #9078. * HW port utils/Python 3: xrange -> range. Re #9078. * Installer/Python 3: xrange -> range. Re #9078. * Keyboard and mouse handlers/Python 3: xrange -> range. Re #9078. * NVDA Ease of Access/Python 3: xrange -> range. Re #9078. * nvwave/Python 3: xrange -> range. Re #9078. * NVDA Python Console/Python 3: xrange -> range. Re #9078. * Queue handler/Python 3: xrange -> range. Re #9078. * Speech viewer, touch tracker/Python 3: xrange -> range. Re #9078. * Virtual buffers/Python 3: xrange -> range, mostly states set construction. Re #9078. * Windows console handler/Python 3: xrange -> range. Re #9078. * Python 3/range: remove source code comments. Review by Reef Turner (NV Access): add explanatory comments for code fragments that are unclear at first. Since xrange to range is trivial and clear enough, reomve such comments. * Speech/Python 3: xrange -> range. Re #9078. * SAPI5 synth driver/Python 3: xrange -> range. Re #9078. * NVDAObjects/UIA/Edge/Python 3: wrap range function used in heading level lookup inside a list. Re #9078. Comment from Leonard de Ruijter (Babbage): there is one genuine need for range function, or at least its Python 2 behavior: EdgeHTML heading lookup, which will lookup heading level text from 1 through 6 inclusive (in a list). Thus wrap this inside a list call. * IAccessible handler/Python 3: add a note about enumerate() function. Re #9078. Comment from Leonard de Ruijter (Babbage): try using enumerate() function to access both the index and the value at hand. Note that this may need to be postponed until this is merged into Project Threshold.
…s expected to fail.
* IAccessible object: except COMError, e -> COMError as e. In Python 2, the exception statement of the form 'except exception, e' was acceptable, no longer the case in Python 3 (requires 'except exception as e'). Thus use the latter in IAccessible objects and others. * Adobe Flash object: except COMError, e -> COMError as e. * Log viewer: except (IOError, OSError), e -> (IOError, OSError) as e. * Input core: except (configobj.ConfigObjError,UnicodeDecodeError), e -> (configobj.ConfigObjError,UnicodeDecodeError) as e. * NVDA slave: except exception, e -> exception as e. * NVWave: except WindowsError, e -> WindowsError as e.
* Update dependencies for Python 3 * Update dependency for py2exe to contain pywin32 * Update submodule references to latest master branches * Update miscDeps to latest master.
* Python 3: import Python 3 native modules. * nvda.pyw: _winreg -> winreg.
* Python 3: os.getcwdu -> os.getcwd. * config.isInstalledCopy: don't catch attributeError when comparing with os.stat. This should never happen. and if it does, we want to know about it. * updateCheck.py: one more use of getcwdu changed.
* Update setup script for Python 3.7 * Fix typo and wrongly formatted tuple * On python 3, Tkinter was renamed to tkinter
* Update batch script and python wrapper for scons * * Update copyright * Remove obsolete _winreg import * No longer pass unicode=True to gettext.install * * sconstruct: xrange>range * comInterfaces_sconscript: iteritems>items * comInterfaces_sconscript: basestring>str * nvdaHelper/archBuild_sconscript: xrange>range * Liblouis sconscript: file>open * cldrDict_sconscript: iteritems>items * Disable developer documentation for now * comInterfaces_sconscript: make sure that the byte compiled targets are picked up correctly by scons * Fix espeak initialization and paths * sconstruct: file>open * Convert doxygen script to Python 3 and fix doxygen site tool * Fix pot files creation * Fix several issues to the dist builder * Issue errors about str(bytes_instance), str(bytearray_instance) * Update readme
* Python 3: Fix use of subprocess * Revert unnecessary change
* Tests/check pot: update copyright years * Tests/check pot: file -> open.
* Bracket some tuples in for loops, and remove L (long-type)suffix from numbers. * Python 3: 'ur' string prefixes no longer supported. Modified 'ur' strings to be just 'r' strings prefixes * More syntax fixes. * Address review comments. * setup.py: exclude win32api which stops py2exe from trying to include pywintypes37.dll. * setup.py: exclude netbios and winxptheme pywin32 modules as they are not needed.
* The logging._levelNames dict no longer exists in Python 3. Instead, we now use logging.getLevelName. * logHandler: We can now use logging.FileHandler rather than logging.StreamHandler and assume unicode all the way until writing the file to disk with utf8. logHandler: FunctionType.func_code is now FunctionType.__code__. # Please enter the commit message for your changes. Lines starting * logHandler: address review comments.
…rly (#9714) * gettext is now always unicode. Insert pgettext into builtins rather than Python 2's __builtin__. * Address review comments. * Unit tests: gettext is unicode.
…test in extension points as expected failure (#9726) * Unittest/Python 3: update copyright years. Re #9720. * Unittest/test driver/Python 3: no more unicode/decoding, proper relative import. Re #9720. * Unittest/base object tests/Python 3: proper relative import. Re #9720. * Unittest/braille tests/Python 3: proper relative import. Re #9720. * Unittest/braille tables/Python 3: dict.itervalues -> dict.values. Re #9720. * Unittest/control types/Python 3: dict.iteritems -> dict.items. Re #9720. * Unittest/text provider/Python 3: just use text as given instead of converting it into Unicode first. Re #9720. * Unittest/script handler/Python 3: assertItemsEqual -> assertCountEqual. Re #9720. Python 2's unittest documentation states self.assertItemsEqual is replaced by self.assertCountEqual in Python 3, thus follow this directive. * Unittest/extension points/Python 3: mark unbound method registration test as expected failure. Re #9720. For some reason unbound method registration test keeps failing in Python 3. Until this is resolved, mark this test as expected failure. * Unittest/base object/Python 3: self.assertRaisesRegexp -> self.assertRaisesRegex due to deprecation warning. Re #9720. * Unittests/review action: remove unnecessary comments. Re #9720. * Unittests/control types: contorlTypes.__dict__ -> vars(contorlTypes). Re #9720. * Extension points/util: add a comment about failing unittest. Re #9720.
* Language list/Python 3: wrap zip function call inside a list call. Behavior of zip() functoin has changed - returning a list in Python 2 versus being an iterator in Python 3. Because language handler/language list uses old zip function behavior, wrap this inside a list call. * IAccessible/IA2mozilla: itertools.izip -> zip, remove itertools import. * Braille: itertools.izip -> zip function. * Language handler: remove extraneous whitespace * Izip to zip (address review actions): simplify comment in language handler, pslit lines in mozilla text iA2 objects.
…#9857) When running a python 2 copy from source, NVDA is unable to detect if it is running under Windows 10. Therefore, the oneCore synthesizer contains a work around in its check method, always listing the synthesizer in the synthesizer list when running from source, even on Windows versions older than Windows 10. On Python 3 however, the manifest of the main Python targets Windows 10 and therefore, this workaround is no longer necessary.
…lag constants (PR #9849) Fixes #9847 Fixes #9825 Add a wrapper for kernel32.dll::MoveFileExW function, along with flag constants. This then allows any module/function/class to use this wrapper instead of defining their own. Re: #9847 Update check: use winKernel.moveFileEx instead of os.renames after postponing an update. Re #9825. Python 3.7's `os.renames` function no longer allows copies across disk drives, it raises `OSError` with ERROR_NOT_SAME_DEVICE. The extended description of this error is "The system cannot move the file to a different disk drive". This can cause an error when updating a portable copy of NVDA" See https://bugs.python.org/issue28356 for information about the change to os.rename / os.renames
…aging Resolved Conflicts: readme.md source/NVDAObjects/IAccessible/__init__.py source/brailleDisplayDrivers/handyTech.py source/gui/settingsDialogs.py source/pythonConsole.py
Searched for, inspected, and fixed usages of "open" function
Used `git grep "\Wopen("` to search of usages of "open"
Fixes #9838
Python can open a file in binary mode and in text mode. In python 2, this difference was less distinct than in Python 3. In python 3, reading a file in text mode returns unicode strings, whereas binary mode returns bytes objects. Therefore, a call of open without supplying proper arguments would open a file in text mode with the default system encoding assumed.
1. #9838: wrong encoding assumed when decoding add-on manifests. As open didn't have an encoding specified, the default text encoding of the system was specified. This failed for Japanese systems.
2. To read Espeak VARIANT DATA, NVDA parses files in the espeak data folder. For this, the open function was also used without an encoding. This resulted in the inability to load espeak on systems where the beta option "Use Unicode UTF-8 for worldwide language support" was enabled.
3. To open calls in NVDAHelper and watchdog are used to open a file without actually writing or reading to it using Python directly. These files are now opened in binary mode. While not strictly necessary, this makes it clearer from a code perspective that we aren't dealing with Python's ability to write or read encoded text files.
4. The logViewer now consistently uses open instead of codecs.open
Fixes #9883 On the Windows 10 lock screen, invoked with Windows+L. The user copy of NVDA is still active. As the screen is locked, there is a set of safe scripts on the appmodule for the lock screen to make sure that the NVDA configuration isn't changed when on the lock screen. However, this SAFE_SCRIPTS set expected scripts on globalCommands.GlobalCommands to be unbound methods, so for every script, it took the __func__ attribute on the unbound method to get to the actual function. In python 3 unbound methods are gone. SAFE_SCRIPTS now just contains the raw scripts from globalCommands.GlobalCommands, which are now functions, no longer unbound methods. When comparing a bound script with the SAFE_SCRIPTS, we're dealing with a bound method and thus we still have to traverse up to the __func__ attribute on the bound script. I added an explanatory comment for that.
…that surrogate characters are handled properly (PR #9897) Before this change, in virtual buffers, emoji are shown as two surrogate characters instead of the actual emoji character. This is because virtual buffers output in UTF-16, and surrogate characters are not allowed in xml. Therefore, the XML contains integer values for the surrogate characters, and they are converted to real characters in the xmlFormatting module. As surrogate characters are perfectly valid within Python 3 and even a surrogate pair is allowed, Python 3 does not collapse two surrogate characters into one 32-bit character. If a low surrogate character is handled, add it to the currently buffered text, and quickly encode and decode the text to/from UTF-16. This ensures that surrogate pairs are properly decoded to the associated 32-bit character. The decision to do this for low surrogates only is intentional. A high surrogate is usually followed by a low surrogate. Therefore it doesn't make sense to re-encode if processing a high surrogate. We also want to avoid just re-encoding anything.
Fixes #9768 The VARIANT out-param was not being initialised correctly. Calling VariantClear on an uninitialised VARIANT is an error. Comtypes was not initialising VARIANT.vt to VT_EMPTY, and attempting to call VariantClear (oleauto.h) before assigning it the value returned from getProp. Instead, we implement a low-level getProp method by requiring an unused "this" param. This low-levl getProp method is provided pointers to the out-params, allowing it to manually initialise the VARIANT correctly before assigning a value.
…9890) These comments are no longer required, they have now been reviewed.
… other identified cases (PR: #9881) Reading documents in Adobe Reader was raising a TypeError Fixed this int to NoneType comparison, identified some others while at it.
Fixes an issue in #9847: catch WindowsError because WinError is not an exception type.
Speech refactor introduced a new way to clear the current words buffer for speech. However, in some cases, this buffer wasn't cleared, particularly in the following cases: - In legacy windows consoles, when typing a word and pressing enter, pressing space still reported the previous word that was already sent to the console. - When moving by sentence in Word, the buffer wasn't cleared. To fix this: - in EditableText, the logic to clear the buffer was moved from _caretMovementScriptHelper to _caretScriptPostMovedHelper. This ensures that the buffer is also cleared when navigating by sentence, while not clearing it when the cart hasn't moved, such as when trying sentence nav in an application that doesn't implement it. - I copied the flush_queuedChars from winConsoleUIA to winConsole. Fixes #9769
Our Python 2 code contained some workarounds and code paths that are no longer necessary or confusing when on Python 3. 1. Most notably, this is related to functions with `*args` and `**kwargs` catch all handlers which also required a specific keyword argument. Once of these was part of the `hwIo` module and has already been removed in an earlier state. - Code where we pop a particular argument from `kwargs` now contains that argument as a keyword only argument in the signature of the function. 2. Furthermore, there was a workaround in the `gui` code to register support for the `cp65001` codec in Python 2, which is a known codec in Python 3. - This is removed 3. The `scriptHandler.script` decorator checked whether the decorated script was a routine. This also returned `True` for bound instance methods, and therefore no warning was raised when trying to decorate a bound instance method (which is unsupported). Now, such a warning will be raised. See also #9884 - When decorating a script, we now use `isinstance(script, types.FunctionType)` 4. `config.AggregatedSection` still had an `iteritems` method. - `config.AggregatedSection.iteritems` has been renamed to items.
NVDA slave module was using the raw_input function to scrape stdin, which raises NameError in Python 3. In Python 3 raw_input has been replaced with the functional equal input. For more information see: https://www.python.org/dev/peps/pep-3111/
Fixes mistake introduced in #9853
We were using range as a name of a local attribute or function parameter. In at least one case (uia _getBoundingRectsFromUIARange), this fails, as were also calling the range function in that method. Looked for every instance of range as a local attribute. In most cases, rename it to textRange.
…aging Conflicts: source/gui/settingsDialogs.py
Contributor
Author
|
To preserve the history, this will not be squash merged. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is intended as a notification of Python 3 support being merged into threshold. Please create any future PR's to fix Python 3 related bugs with threshold as a base branch.
If threshold seems mostly stable, we will merge to master (approximate time frame: 3-5 days)