locale.setlocale raising exception

[laser-axel]

Problem

cannot import ZOSPy 1.2.0 as is.
It is raising a locale.Error: unsupported locale setting

What version of ZOSPy are you running?

1.2.0

What version of OpticStudio are you running?

2023 R1.03

Which operating system do you use?

Windows 10

In which environment do you use ZOSPy?

• Plain Pythons scripts (not in Spyder)
• Jupyter notebooks
• Spyder

In which connection modes does the problem occur?

• Extension mode
• Standalone mode

Example code

loc = locale.getlocale()
# in ZOSPy's __init__.py there is also a
# locale.setlocale(locale.LC_ALL, "") and some things happen to make ZOSPy aware of the user preferences
print(loc)
locale.setlocale(locale.LC_ALL, loc)

Output

('de_DE', 'cp1252')
Traceback (most recent call last):
  File "C:\Users\XXXXX\Documents\Python Scripts\locale-pain.py", line 11, in <module>
    locale.setlocale(locale.LC_ALL, loc)  # restore saved locale
  File "C:\Users\XXXXX\Anaconda3\lib\locale.py", line 610, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

Additional information

No response

[crnh]

Hi,

Thanks for reporting this! locale.getlocale() appears to not always return a valid (RFC1766) language code. According to this issue the locale settings in Windows are "a hot mess and it won't be solved anytime soon." This might also cause your problems.

Question: what happens if you run this code prior to importing ZOSPy:

import locale
locale.setlocale(locale.LC_ALL, "")

Do you then still get an exception on import (I am afraid you will)?

I looked up why we implemented the locale options in zospy.api.config this way because I thought we might be able to get away with determining the decimal point etc. without first activating the default locale (locale.setlocale(LC_ALL, "")). Turns out this won't work: on my PC, the decimal separator in locale.localeconv() is incorrect if I do not run locale.setlocale(LC_ALL, "") first.

It is possible to not reset the locale to the original locale after determining the decimal separators etc., but then we may permanently change the locale for the full script in which ZOSPy is used, and that is something I prefer not to do.

[laser-axel]

Python 3.11.7 | packaged by Anaconda, Inc. | (main, Dec 15 2023, 18:05:47) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, "")
'German_Germany.1252'
>>> import zospy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\aheinrici\Anaconda3\envs\zospy\Lib\site-packages\zospy\__init__.py", line 12, in <module>
    config.set_decimal_point_and_thousands_separator()
  File "C:\Users\aheinrici\Anaconda3\envs\zospy\Lib\site-packages\zospy\api\config.py", line 30, in set_decimal_point_and_thousands_separator
    locale.setlocale(locale.LC_ALL, loc)  # restore saved locale
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\aheinrici\Anaconda3\envs\zospy\Lib\locale.py", line 627, in setlocale
    return _setlocale(category, locale)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
locale.Error: unsupported locale setting
>>>

I partly understand that Zemax is using commas in the GUI on German localizied Systems. But for programming I think it would be OK not to interfere with the locale at all.
Question is the relevancy of the locale's decimal separator. The only way I can think where this could be relevant, is when ZOSPy is taking care of the output formatting.
Anyway, Inside Python I have to use dots in float literals.

>>> import math, locale
>>> locale.getlocale()
('de_DE', 'cp1252')
>>> print(math.pi)
3.141592653589793

And Python uses dots in output.
And when using any sort of API, I would expect numbers to come in as values not string.

Optic Studio itself is putting commas into text output on german localized systems, but it only accepts dots in text input (i.e. source files)

[crnh]

Hi,

Thanks for checking! ZOSPy uses the decimal and thousands separators to parse the text output of analyses. Many OpticStudio analyses do not provide access to their results through the ZOS-API, so you need to use the text output (you can check the Zernike Standard Coefficients analysis for an example). Knowing the decimal separator is very relevant for that purpose. In the past we used regexes that allowed for both a dot and a comma as decimal separator, but this turned out to be unreliable in some cases. If you want to know more about this, please take a look at the other issues / pull requests in this repository.

Another question: what happens if you replace this line in the original zospy/api/config.py:

ZOSPy/zospy/api/config.py

Line 26 in b9cef91

loc = locale.getlocale() # get and save current locale

with

loc = locale.setlocale(locale.LC_ALL)

Do you still get a locale.Error?

[laser-axel]

Yes, that makes the error on importing disappear.
Beeing more explicit, I also tried
loc = locale.setlocale(locale.LC_ALL, locale=None)
which behaves the same.
Put some logging lines afterwards and I can see that DECIMAL_POINT becomes "," and THOUSANDS_SEPARATOR ".".

That seems to be correct. But Zernike Standard Coefficients analysis is not working.

ValueError                                Traceback (most recent call last)
Cell In[10], line 1
----> 1 zospy.analyses.wavefront.zernike_standard_coefficients(oss)

File ~\Anaconda3\envs\zospy\Lib\site-packages\zospy\analyses\wavefront.py:173, in zernike_standard_coefficients(oss, sampling, maximum_term, wavelength, field, reference_opd_to_vertex, surface, sx, sy, sr, oncomplete, txtoutfile)
    170 analysis.Results.GetTextFile(txtoutfile)
    171 line_list = [line for line in open(txtoutfile, "r", encoding=oss._ZOS.get_txtfile_encoding())]
--> 173 general_data, zernike_data = _structure_zernike_standard_coefficients_result(line_list)
    174 data = AttrDict(GeneralData=general_data, Coefficients=zernike_data)
    176 # Get headerdata, metadata and messages

File ~\Anaconda3\envs\zospy\Lib\site-packages\zospy\analyses\wavefront.py:56, in _structure_zernike_standard_coefficients_result(line_list)
     54 elif len(dat) == 1:
     55     if valuepat_start.search(dat[0].replace(",", ".")):  # value is number
---> 56         val = float(dat[0].replace(",", "."))
     57     else:
     58         val = dat[0]

ValueError: could not convert string to float: '07.03.2024'

--Test was run on 2024-03-07

As far as I can see,
Zernike Standard Coefficients analysis) uses encoding=oss._ZOS.get_txtfile_encoding() which is utf16-little-endian to create line_list. line_list ist just a list of strings.
line_list goes into _structure_zernike_standard_coefficients_result(line_list) which uses regexes on outputs of str.replace(",", ".") to decode that stuff. I don't recognize where the locale setting is becoming relevant. The error seems to be related with _structure_zernike_... trying to interpret current date as data.

[crnh]

I see, the Zernike Standard Coefficients is an older analysis that was written in the early stages of ZOSPy. It doesn't use the localized decimal separator yet. GitHub has a search function (your IDE has one, too) so it is quite easy to search for uses of symbols. If you use that search function you will see that some newer analyses (in polarization.py and raysandspots.py) do use these settings.

You just identified another issue, which is related to date notation. I didn't encounter this date notation earlier and suppose @LucVV also missed this case. Anyways, this is exactly the reason why we are re-implementing the analysis parsers from scratch. I agree with one of your previous comments that all analysis results should be available through the API. Unfortunately, Zemax / Ansys doesn't seem to be working on this or even considering this, so it is still useful to have parsers in ZOSPy.

Thanks for testing my suggestion, good to see it works. We are currently working on a solution.

[jwmbeenakker]

I propose we go for locale.setlocale solution, as it works and is also used in the Python standard library:
https://github.com/python/cpython/blob/5c69f60ae1859e53f858e65636d1c6d83873a963/Lib/locale.py#L682C9-L690C41

[crnh]

A fix to this problem has just been released with ZOSPy 1.2.1.