encoding issue when using GRASS processing tool on macOS

[daveism]

I tried this on two different Macs with two different OS versions: Mojave and Catalina.
Als tried this with two different versions of QGIS: 3.18.0-Zürich and 3.16.4-Hannover

When I run any shapefile or geopackage or any GIS bases file with the projection = epsg:3857 any GRASS processing tool fails to run due to Unicode error

Traceback (most recent call last):
File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/plugins/processing/algs/grass7/Grass7Algorithm.py", line 423, in processAlgorithm
getattr(self, fullName)(parameters, context, feedback)
File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/plugins/processing/algs/grass7/Grass7Algorithm.py", line 490, in processInputs
self.loadVectorLayerFromParameter(
File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/plugins/processing/algs/grass7/Grass7Algorithm.py", line 877, in loadVectorLayerFromParameter
self.loadVectorLayer(name, layer, external=external, feedback=feedback)
File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/plugins/processing/algs/grass7/Grass7Algorithm.py", line 914, in loadVectorLayer
self.setSessionProjectionFromLayer(layer)
File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/plugins/processing/algs/grass7/Grass7Algorithm.py", line 1044, in setSessionProjectionFromLayer
self.setSessionProjection(layer.crs())
File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/plugins/processing/algs/grass7/Grass7Algorithm.py", line 1051, in setSessionProjection
file_name = Grass7Utils.exportCrsWktToFile(crs)
File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/plugins/processing/algs/grass7/Grass7Utils.py", line 95, in exportCrsWktToFile
f.write(wkt)
UnicodeEncodeError: 'ascii' codec can't encode character '\xb0' in position 849: ordinal not in range(128)

which is the degree symbol in the usage part of the WKT

As a note when I use the same shapefile or geopackage on the same machine using QGIS 3.8.0-Zanziba the grass processing tools runs without any issue, on both Mac's.

all installers were from dmg's downloaded from QGIS website.

[roya0045]

Declaring an encoding on

QGIS/python/plugins/processing/algs/grass7/Grass7Utils.py

Line 94 in 0a3b7d3

with open(wkt_file, 'wt') as f:

should do the trick. If you edit your own python files to add encoding='utf-8' to get with open(wkt_file, 'wt', encoding='utf-8' ) as f: does it fix it?

[daveism]

I thought it would be easy, and that was the first thing I tried, but it did not work. In fact, I tried many variations of encoding and trying force encoding='utf-8'

In the end, I had to open the file in binary write. For whatever reason, nothing else worked. I also had to encode as ASCII and ignore the errors. I tried everything else. It makes no sense to me. It's probably worth digging into further if I had the time.

        with open(wkt_file, 'wb') as f:
            f.write(wkt.encode("ascii", "ignore"))

Either way might be something to include in the QGIS downloads, so others don't run into the same issue, especially since EPSG:3857 gets used quite a bit.

[roya0045]

strange as the error clearly point to the degree symbol in utf-8.

[daveism]

I won't pretend to understand why this did not work with open(wkt_file, 'wt', encoding='utf-8' )

I removed QGIS and even reinstalled local versions of GDAL. That's when I tried it on both my personal and work Mac with the same result—I assumed I was doing something wrong. I am still not convinced that I am not doing something wrong because it makes 0 sense. What did work was QGIS 3.8.0-Zanziba build on the same machines. I did not look in detail, but I am guessing at some point a GDAL update added the use when (I think that was the place the degree character was added).

Anyway, It's working for me now, and I give up 😞 on understanding.

[gioman]

Anyway, It's working for me now, and I give up disappointed on understanding.

@daveism is this data dependent? if yes can you please attach a sample?

[daveism]

@gioman not data-dependent but definitely projection-dependent. Any dataset that had the projection epsg:3857 defined resulted in the same error. I tried several different geom types (point, line, poly).

For sanity checks, I also created completely new shapefiles, geopackages, and even PostGIS layers. Anytime the layer had the projection defined as epsg:3857—error. I tested with v.buffer, v.clean, and v.net.path (what I was trying to run). I also ran a few grass raster processes and had the same error. (I forget the specific ones)

When I reprojected the same files into epsg:4326 or EPSG:2264 because I live in North Carolina, no errors.

I suspected that the GDAL paths on my work computer got crossed somehow. I tried my personal computer, which has only one GDAL version setup. When I saw the same errors, I figured it was some kind of bug, and I should report it.

[gioman]

@daveism no issues on Linux and Windows

I also ran a few grass raster processes and had the same error. (I forget the specific ones)

Do you mean that you have the same issue when you use directly native GRASS (so not using QGIS/Processing of the QGIS GRASS plugin)? If yes then you have an issue with GRASS in your installation.

I will check anyway on macOS as soon as possible.

[daveism]

did not try Grass natively or via the plugin.

I was able to replicate the error in the QGIS python console with this:

crs = QgsCoordinateReferenceSystem('EPSG:3857')
wkt_file = QgsProcessingUtils.generateTempFilename('crs.prj')
wkt = crs.toWkt(QgsCoordinateReferenceSystem.WKT_PREFERRED)
with open(wkt_file, 'wt') as f:
    f.write(wkt)

which means it really had nothing to do with GRASS

I tried with open(wkt_file, 'wt', encoding='utf-8' ) on python console and it worked but when I did the same in QGIS/python/plugins/processing/algs/grass7/Grass7Utils.py the error kept happening. not sure why

only when I changed QGIS/python/plugins/processing/algs/grass7/Grass7Utils.py to

        with open(wkt_file, 'wb') as f:
            f.write(wkt.encode("ascii", "ignore"))

was able to run grass processing tools

[gioman]

crs = QgsCoordinateReferenceSystem('EPSG:3857')
wkt_file = QgsProcessingUtils.generateTempFilename('crs.prj')
wkt = crs.toWkt(QgsCoordinateReferenceSystem.WKT_PREFERRED)
with open(wkt_file, 'wt') as f:
f.write(wkt)

@daveism If I should get an error after the above snippet then I can't confirm on Linux and Windows.

which means it really had nothing to do with GRASS

Can you please edit the title and description to better describe your latest findings? thanks.

[github-actions]

The QGIS project highly values your report and would love to see it addressed. However, this issue has been left in feedback mode for the last 14 days and is being automatically marked as "stale".
If you would like to continue with this issue, please provide any missing information or answer any open questions. If you could resolve the issue yourself meanwhile, please leave a note for future readers with the same problem and close the issue.
In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this issue.
If there is no further activity on this issue, it will be closed in a week.

[Shelomoh1]

I have the same issue with QGIS 3.16 LTR and 3.18. All GRASS processing toolbox modules give the same error. Any fix yet? @gioman @daveism

[daveism]

@Shelomoh1, the fix for me was to edit the file QGIS/python/plugins/processing/algs/grass7/Grass7Utils.py, which you can find in the QGIS installation. The location of the file will on depend on your OS.

For my installation, this line and the preceding line was the culprit #41870 (comment)

To resolve the issue on my machine, I had to change the python code on both lines to match this:

        with open(wkt_file, 'wb') as f:
            f.write(wkt.encode("ascii", "ignore"))

Once I made the edit and completely exited QGIS, everything worked as expected.

I am not sure what it will take to update this in the source code since I am not a contributor; I just reported the issue and what I did to fix it.

[gioman]

@daveism have you seen #41870 (comment) ?

[daveism]

@gioman yes it sounded like you were going to check if you got the error on Windows and Linux. I don't have access to either OS if you were asking me to do it.

I am not sure what you think the title and description should be changed to better describe my latest findings, let me know what you think is missing and I am happy to change it.

[Shelomoh1]

@Shelomoh1, the fix for me was to edit the file QGIS/python/plugins/processing/algs/grass7/Grass7Utils.py, which you can find in the QGIS installation. The location of the file will on depend on your OS.

For my installation, this line and the preceding line was the culprit #41870 (comment)

To resolve the issue on my machine, I had to change the python code on both lines to match this:

        with open(wkt_file, 'wb') as f:
            f.write(wkt.encode("ascii", "ignore"))

Once I made the edit and completely exited QGIS, everything worked as expected.

I am not sure what it will take to update this in the source code since I am not a contributor; I just reported the issue and what I did to fix it.

I tried the editing the code following your example and restarted QGIS but the same error appears. My path is "/Applications/QGIS3.18.app/Contents/Resources/python/plugins/processing/algs/grass7/Grass7Utils.py" on Macos Catalina.

[daveism]

I was on High Sierra so maybe its something different for Catalina

The issue is around the degree symbol in the WKT output from GDAL. When opening the file for write you will need to do something for your installation that encodes the degree symbol so it can be written to the WKT file.

To troubleshoot try opening the python console in QGIS. Start with this.

crs = QgsCoordinateReferenceSystem('EPSG:3857')
wkt_file = QgsProcessingUtils.generateTempFilename('crs.prj')
wkt = crs.toWkt(QgsCoordinateReferenceSystem.WKT_PREFERRED)
with open(wkt_file, 'wt') as f:
    f.write(wkt)

and mess around with these two lines

with open(wkt_file, 'wt') as f:
    f.write(wkt)

until it works.

Then you can edit /Applications/QGIS3.18.app/Contents/Resources/python/plugins/processing/algs/grass7/Grass7Utils.py and things should work for you too.

[Shelomoh1]

I was on High Sierra so maybe its something different for Catalina

The issue is around the degree symbol in the WKT output from GDAL. When opening the file for write you will need to do something for your installation that encodes the degree symbol so it can be written to the WKT file.

To troubleshoot try opening the python console in QGIS. Start with this.

crs = QgsCoordinateReferenceSystem('EPSG:3857')
wkt_file = QgsProcessingUtils.generateTempFilename('crs.prj')
wkt = crs.toWkt(QgsCoordinateReferenceSystem.WKT_PREFERRED)
with open(wkt_file, 'wt') as f:
    f.write(wkt)

and mess around with these two lines

with open(wkt_file, 'wt') as f:
    f.write(wkt)

until it works.

Then you can edit /Applications/QGIS3.18.app/Contents/Resources/python/plugins/processing/algs/grass7/Grass7Utils.py and things should work for you too.

Okay. Let me try that as well and see the outcome. Thank you very much.

[daveism]

@Shelomoh1 One thing I just remembered. I had to do a force quit of QGIS to get QGIS to release whatever it had cached for my change in the Grass7Utils.py file to take.

[Shelomoh1]

@Shelomoh1 One thing I just remembered. I had to do a force quit of QGIS to get QGIS to release whatever it had cached for my change in the Grass7Utils.py file to take.

Well noted. Thank you once again.

[gioman]

@gioman yes it sounded like you were going to check if you got the error on Windows and Linux. I don't have access to either OS if you were asking me to do it.

I am not sure what you think the title and description should be changed to better describe my latest findings, let me know what you think is missing and I am happy to change it.

@daveism I can see the error being referred here when I run this in the pyhton console on macOS Catalina and QGIS 3.16.4

crs = QgsCoordinateReferenceSystem('EPSG:3857')
wkt_file = QgsProcessingUtils.generateTempFilename('crs.prj')
wkt = crs.toWkt(QgsCoordinateReferenceSystem.WKT_PREFERRED)
with open(wkt_file, 'wt') as f:
    f.write(wkt)

but this apart the GRASS tools in the Processing toolbox work ok for me, also when using layers with EPSG 3857 as input.

[daveism]

@gioman it sounds like others are having an issue with EPSG 3857 as input, I am not a contributor to QGIS nor do I understand how I can become one. Is this something that I need to create a pull request for? I reported the issue and it's not clear what action needs to happen or is expected to get this fixed.

[roya0045]

@daveism Fork the repo, in your branch make the changes to the file, and then submit the PR. There are a few guides online. Should be relatively painless.

[Shelomoh1]

@Shelomoh1 One thing I just remembered. I had to do a force quit of QGIS to get QGIS to release whatever it had cached for my change in the Grass7Utils.py file to take.

Well noted. Thank you once again.

@daveism I tried again and this time it worked. The algorithm was able to produce the results. However, when I check the process log, there is another error while exporting the output to Geotiff. I hope that's no cause for alarm.

[roya0045]

@Shelomoh1 that might be a GRASS, unrelated to the current issue. You could change your data type to int16 if this doesn't make you lose information.

[Shelomoh1]

@Shelomoh1 that might be a GRASS, unrelated to the current issue. You could change your data type to int16 if this doesn't make you lose information.

The data type is already int16. Anyway, I run other grass algorithms and that error didn't appear so I guess you're right.

[boillodmanuel]

If encoding is not set explicitly in open, I think it relies on the system locale defined by the LC_ALL environment variable.

Here is my tests (run in QGIS python console):

# Explicitly open with ascii encoding:
# ❌ Error:
#   UnicodeEncodeError: 'ascii' codec can't encode character '\u2122' in position 0: ordinal not in range(128)
with open('/tmp/test', 'w', encoding='ascii') as fout:
    fout.write(u"\u2122")

# Open without encoding specified:
# ❌ Error or ✅ Works depending of system locale :
#   UnicodeEncodeError: 'ascii' codec can't encode character '\u2122' in position 0: ordinal not in range(128)
with open('/tmp/test', 'w') as fout:
    fout.write(u"\u2122")
    
# Explicitly open with utf-8 encoding:
# ✅ Works
with open('/tmp/test', 'w', encoding='utf-8') as fout:
    fout.write(u"\u2122")

To force the system locale in QGIS, I added the LC_ALL environment variable in preferences.

• Open QGIS > Preferences > System
• Check "Use custom variable"
• Add LC_ALL = en_US.UTF-8 or whatever you want.
  I'm not sure we should keep this settings, but it helps to debug.

I tried to defined this variable in my bash_profile but it didn't work.

Env: I'm using macos bigsur 11.2.3

[nyalldawson]

@PeterPetrik is this an issue in the mac packaging?

[PeterPetrik]

it looks like it could relate to linux too. Looks like the computer locale vs grass/gdal tools.

[beedawg1985]

@Shelomoh1, the fix for me was to edit the file QGIS/python/plugins/processing/algs/grass7/Grass7Utils.py, which you can find in the QGIS installation. The location of the file will on depend on your OS.

For my installation, this line and the preceding line was the culprit #41870 (comment)

To resolve the issue on my machine, I had to change the python code on both lines to match this:

        with open(wkt_file, 'wb') as f:
            f.write(wkt.encode("ascii", "ignore"))

Once I made the edit and completely exited QGIS, everything worked as expected.

I am not sure what it will take to update this in the source code since I am not a contributor; I just reported the issue and what I did to fix it.

This worked for me on QGIS-LTR 3.16, Big Sur 11.1.

[rideearthtom]

To force the system locale in QGIS, I added the LC_ALL environment variable in preferences.

* Open QGIS > Preferences > System

* Check "Use custom variable"

* Add `LC_ALL` = `en_US.UTF-8` or whatever you want.
  I'm not sure we should keep this settings, but it helps to debug.

This solved the issue for me. Big Sur 11.2.3 and QGIS 3.18.

[gioman]

This solved the issue for me. Big Sur 11.2.3 and QGIS 3.18.

made a clean installation of 3.18.3 on a clean Catalina machine, hit the same problem and this workaround also fixed it for me.

[PeterPetrik]

@rouault Do you know if .prj file containing WKT definition CAN contain non-ASCII chars? It looks here we want to save there \xb0 (degree sign) for epsg:3857 as it all comes from

QGIS/python/plugins/grassprovider/Grass7Utils.py

Line 92 in adc9d07

wkt = crs.toWkt(QgsCoordinateReferenceSystem.WKT_PREFERRED)

It works on computers with LC_ALL = en_US.UTF-8.

I am tempting to just add encoding='utf-8' to the Python open that tries to save the string to the file, but wan't to check if it is reasonable fix.

Thanks!

[rouault]

It depends... The only well established specification for .prj file is when using ESRI WKT for shapefiles, which is I far as I known ASCII only. All other uses (for other formats, or using non ESRI WKT as here) are extensions. UTF-8 content can certainly come from WKT 2, as output by PROJ. In that context, it is probably ok using encoding='utf-8'