QGIS3/GRASS: error while executing a script

[fjperini]

Problem reported by @luisspuerto

Your PYTHONPATH points to a site-packages dir for Python 3.x but you are running Python 2.x!

PYTHONPATH is currently: "/usr/local/opt/gdal2-python/lib/python3.7/site-packages:/usr/local/opt/qgis3/lib/python3.7/site-packages:/usr/local/opt/qgis3/libexec/python/lib/python3.7/site-packages:/usr/local/lib/python3.7/site-packages"

You should `unset PYTHONPATH` to fix this.

[luispuerto]

I've built Grass7 form source using this formula and rebuilt qgis3 and the problem is still there.

[luispuerto]

Where should I change that you mention on
#436 (comment), sorry?

[fjperini]

@luisspuerto I think that default_gui = "text" it seems to be a GRASS variable and #!/usr/bin/env python3 in each GRASS script. We can define that variable in the formula.

GRASS startup program
GRASS variables and environment variables

[luispuerto]

This isn't fixed by the last merge #446, is it?

[fjperini]

@luisspuerto I have not reviewed this problem. Can you test the above?
I think another option could be to add PYTHONPATH in the script: g.gisenv set="VARIABLE=VALUE" Example: GRASS_GUI=text, GRASS_PYTHON=3
Will work this PYTHONPATH=/usr/local/lib/python2.7/site-packages?

[luispuerto]

After the problems I report in #456, I've tried to run a grass script and I get the following:

sys.stdout.write("%s\n" % grass_version)

^

TabError: inconsistent use of tabs and spaces in indentation

File "/usr/local/Cellar/grass7/7.4.1/libexec/bin/grass74", line 1693

sys.stdout.write("%s\n" % grass_version)

^

TabError: inconsistent use of tabs and spaces in indentation

Execution completed in 0.69 seconds

Here the full output.

[luispuerto]

By the way after the rebuilding of qgis3 —I don't know how many times— I've noticed the following lines...

NOTE: Your current PATH and PYTHONPATH environment variables are honored
      when launching via the wrapper script, while launching QGIS.app
      bundle they are not.

For standalone Python3 development, set the following environment variable:
  export PYTHONPATH=/usr/local/opt/qgis3/lib/python3.7/site-packages:/usr/local/opt/gdal2-python/lib/python3.7/site-packages:/usr/local/opt/qgis3/libexec/vendor/lib/python3.7/site-packages:$PYTHONPATH

[nickrobison]

@luisspuerto The tab/space issue is an upstream problem in grass, I think it was first reported in #228. I'll try and create a patch and upgrade the grass formula.

[luispuerto]

In qgis2 everything works! thankgod!

[luispuerto]

I just updated to QGIS3 and QGIS2. In QGIS2 everything seems to work perfectly, but in QGIS3 the error is still there... 😞

[fjperini]

@luisspuerto, the problem is in GRASS, @nickrobison will create a patch to solve it.

[fjperini]

@luispuerto, is this still happening? If it is solved we close this issue

[luispuerto]

@fjperini No... sadly isn't working... but I think it's different error now:

Processing algorithm…
Algorithm 'v.buffer' starting…
Input parameters:
{ '-c' : False, '-s' : False, '-t' : False, 'GRASS_MIN_AREA_PARAMETER' : 0.0001, 'GRASS_OUTPUT_TYPE_PARAMETER' : 0, 'GRASS_REGION_PARAMETER' : None, 'GRASS_SNAP_TOLERANCE_PARAMETER' : -1, 'GRASS_VECTOR_DSCO' : '', 'GRASS_VECTOR_LCO' : '', 'angle' : 0, 'cats' : '', 'column' : None, 'distance' : 1000, 'input' : '/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data_modified/Geomorphology/grens_waterschap.shp', 'minordistance' : 10, 'output' : '/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_421a71786e17438da2dd7f2f345db0ea/1170294241af4750a49e0ae27b196c73/output.shp', 'scale' : 1, 'tolerance' : 0.01, 'type' : [0,1,4], 'where' : '' }

g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"
v.in.ogr min_area=0.0001 snap=-1.0 input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data_modified/Geomorphology/grens_waterschap.shp" output="vector_5bdd689130a3b10" --overwrite -o
g.region n=503941.924 s=461701.483 e=269799.23 w=222306.806
v.buffer input=vector_5bdd689130a3b10 type="point,line,area" distance=1000 minordistance=10 angle=0 scale=1 tolerance=0.01 output=output327607dda92d44cf87efbd523b81e66e --overwrite
v.out.ogr -c type="auto" input="output327607dda92d44cf87efbd523b81e66e" output="/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_421a71786e17438da2dd7f2f345db0ea/1170294241af4750a49e0ae27b196c73/output.shp" format="ESRI_Shapefile" --overwrite
File "/usr/local/Cellar/grass7/7.4.2/libexec/bin/grass74", line 1693
sys.stdout.write("%s\n" % grass_version)
^
TabError: inconsistent use of tabs and spaces in indentation
File "/usr/local/Cellar/grass7/7.4.2/libexec/bin/grass74", line 1693
sys.stdout.write("%s\n" % grass_version)
^
TabError: inconsistent use of tabs and spaces in indentation
Execution completed in 0.66 seconds
Results:
{'output': <QgsProcessingOutputLayerDefinition {'sink':/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_421a71786e17438da2dd7f2f345db0ea/1170294241af4750a49e0ae27b196c73/output.shp, 'createOptions': {'fileEncoding': 'System'}}>}

Loading resulting layers
The following layers were not correctly generated.<ul><li>/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_421a71786e17438da2dd7f2f345db0ea/1170294241af4750a49e0ae27b196c73/output.shp</li></ul>You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.

grass log window:

2018-11-03T11:24:03     INFO    processInputs end. Commands: ['g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"', 'v.in.ogr min_area=0.0001 snap=-1.0 input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geomorphology/grens_waterschap.shp" output="vector_5bdd69339c08d2" --overwrite -o', 'g.region n=503941.924 s=461701.483 e=269799.23 w=222306.806']
2018-11-03T11:24:03     INFO    processCommands end. Commands: ['g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"', 'v.in.ogr min_area=0.0001 snap=-1.0 input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geomorphology/grens_waterschap.shp" output="vector_5bdd69339c08d2" --overwrite -o', 'g.region n=503941.924 s=461701.483 e=269799.23 w=222306.806', 'v.buffer input=vector_5bdd69339c08d2 type="point,line,area" distance=1000 minordistance=10 angle=0 scale=1 tolerance=0.01 output=output32e22da1751f43b3a65a38768a715e77 --overwrite']

In QGIS2 works.

I also tested other script in grass I the same/similar error.

[luispuerto]

PS/ by the way all the others works perfectly, including saga, que didn't work before.

orfeo6 disappeared again, but I guess you still working in implementation, don't you?

[luispuerto]

I just install QGIS 3.4.1 and the error still happening... I guess it's a QGIS issue.

[luispuerto]

Additional info in the geoprocessing log:

2018-11-03T17:29:07     INFO    GRASS GIS 7 execution commands
             g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"
             v.external input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/bedekkingpleistoceen.shp" output="vector_5bddbec33f9fb2" --overwrite -o
             g.region n=500000.0 s=461702.4058968832 e=269723.15625 w=230968.3125
             v.buffer input=vector_5bddbec33f9fb2 type="point,line,area" distance=100 angle=0 scale=1 tolerance=0.01 output=outpute32856ce1f2042d58ece8ab94ba33b2b --overwrite
             v.out.ogr -c type="auto" input="outpute32856ce1f2042d58ece8ab94ba33b2b" output="/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_0cb9d254b42a4b6b8ad15f38c86c2294/2416a9a64bf74f0a94a8fb03288f1edb/output.shp" format="ESRI_Shapefile" --overwrite
2018-11-03T17:29:07     INFO    GRASS GIS 7 execution console output
              File "/usr/local/Cellar/grass7/7.4.2/libexec/bin/grass74", line 1693
             
              sys.stdout.write("%s\n" % grass_version)
             
              ^
             
             TabError: inconsistent use of tabs and spaces in indentation
             
             File "/usr/local/Cellar/grass7/7.4.2/libexec/bin/grass74", line 1693
             sys.stdout.write("%s\n" % grass_version)
             ^
             TabError: inconsistent use of tabs and spaces in indentation

TabError: inconsistent use of tabs and spaces in indentation

Well... I don't know. I don't know why this happens in QGIS3 but not in 2.

[fjperini]

@luispuerto This commented @nickrobison in #228

@bolliger32 That's a really strange error, I can see the incorrect indenting in my grass74 file, but no error in thrown.

Which version of python are you using?

If you remove the tabs at the beginning on the line, and ensure that the indentation is lined up correctly, does the error still occur?

Regarding the problem with PYTHON, I noticed that in src/plugins/grass/qgsgrassshell.cpp:

env << QStringLiteral( "GRASS_PYTHON=python" );

It is very likely that you are calling python3 by default when grass is configured to use python2

Another thing you can do to make sure:

export GRASS_PYTHON=$(brew --prefix)/opt/python2/bin/python2

GRASS addons may fail due to python3 incompatibility. If that is tha case you can change the shebang a the beginning of the script to enforce python2 usage.

#!/usr/bin/env python

Should be changed into

#!/usr/bin/env python2

Try this and then tell me how it turned out!

p/d: Yes, I am working on the implementation of orfeo6, I applied several changes with respect to #461 #462.

[luispuerto]

@fjperini I can try to change python 3 for 2 for grass, but I really don't know how to do it inside the QGIS3 app. I really understand about the shebang, but I'm not scripting, I'm trying to use just the v.buffer script form the grass toolbox in QIGS3.

[fjperini]

@luispuerto I get it!
I will sit with this.

[fjperini]

@luispuerto @nickrobison It seems that grass7 works in qgis3! 🎉

I had to apply several changes.
Other errors, I solved it with:

---

For the error "ValueError: unknown locale: UTF-8" in /qgis3/libexec/vendor/lib/python3.7/locale.py

I think this error occurs when QGIS starts with another LANG other than en_US.UTF-8.

Check what kind of local variables:

$ ls /usr/share/locale/

I added it to /qgis3/3.4.2/bin/qgis3 (only if it is executed from console), it is better to place it in ~/.bash_profile

export LANG=en_US.UTF-8
export LC_CTYPE=en_US.UTF-8
export LC_ALL=en_US.UTF-8

$ source  ~/.bash_profile

$ python -c 'import locale; print(locale.getdefaultlocale());'

('en_US', 'UTF-8')

UPDATE:

It is not enough to put it in ~/.bash_profile and /qgis3/3.4.2/bin/qgis3, it is better to put it in /grass7/7.4.3/libexec/bin/grass74 for a definitive solution.

---

For the error TypeError("str expected, not %s" % type(value).name) / TypeError: str expected, not bytes in /grass7/7.4.3/libexec/bin/grass74

I had to change the next line

os.environ['MANPATH'] = sys_man_path

to

  os.environ['MANPATH'] = str(sys_man_path)

---

p/d: I hope it works, at least I was not presented with the errors that were previously.

[luispuerto]

😱 this is grasstantic I can wait to test it!

[fjperini]

It is! First we will have to merge qt5-webkit, then I apply the changes for grass7 and qgis3.

Grasstantic 😄 Well said!

[fjperini]

As mentioned @msandifo (see), a solution:

a temp fix is to replace tabs with spaces by editing line 1693 in /usr/local/Cellar/grass7/7.4.1/libexec/bin/grass74

which can be applied by @nickrobison #549 :

# Patch grass.py to remove bad tab at line 1693 (and insert 12 spaces)
# Needs to be pushed upstream.
inreplace "lib/init/grass.py", "\t\t\t", " 

or, with the patch that was to apply:

--- a/lib/init/grass.py
+++ b/lib/init/grass.py
@@ -1668,7 +1668,7 @@ def print_params():
             filerev.close()
             sys.stdout.write("%s\n" % val[0].split(':')[1].rstrip('$"\n').strip())
         elif arg == 'version':
-			sys.stdout.write("%s\n" % grass_version)
+            sys.stdout.write("%s\n" % grass_version)
         else:
             message(_("Parameter <%s> not supported") % arg)

Once merged #549, I will apply the other changes.

@luispuerto When the bottle for qt5-webkit is ready, I to apply changes in qgis3 and improve integration with GRASS, R and ORFEO.

[fjperini]

@luispuerto I'm about to upload the other changes!

[fjperini]

@luispuerto You can try the changes, in the course of the day I apply to qgis3.

They are all integrated and functioning.

[luispuerto]

Great! @fjperini are the changes related to the last 3 pull request? I can download them and merge locally in a branch and test them

[fjperini]

@luispuerto We finally beat GRASS. At some point you'll have to give me that beer. Ha!

Yes, it is necessary to delete the previous profile to avoid inconveniences.

Keep testing and then tell me.

[luispuerto]

@fjperini where do you live? if you live in Spain perhaps we have a chance this Christmas

[fjperini]

@luispuerto We are far, I to a sea. Ha! But soon I will be walking through the old continent.

The integration of R and ORFEO is completed. It will only be necessary to enable them from Manage and Install Plugins and and ready.

I was workining to configure the default folders and activate the plugin when it is enabled:

Manage and Install Plugins -> Installed -> Plugin name (click its checkbox)

I correct the --with-globe integration and apply the changes. With that would be all the corrections to supports that were by default.

[nickrobison]

@fjperini @luispuerto I would be more than happy to buy you each a beer this Christmas, if you let me know where to send the money! One for @luispuerto for finding all the errors, and one for @fjperini for fixing them!

[luispuerto]

@nickrobison as far I see you are based in Seattle, perhaps I will go around the great Pacific North West after the summer for a couple of months. 🍻 ?

PS/ You have a lot of trees around there 🌲 🌲 🌲 🌲 🌲 🌲 🌲 🌲 I'm a forester.

[fjperini]

@luispuerto Did you verify that GRASS works 100%? To confirm that it was resolved and close this issue.

@nickrobison With the R, ORFEO and GLOBE resolved, it will have to be a full beer box. What do we do, do we demand it? 😄

We will continue the talk here #431 for the integration of ORFEO. I finish testing the --with-globe and apply the changes.

p/p: It would be great to be able to meet!

[luispuerto]

@fjperini I run a couple of scripts more... but they aren't working.

this is the log while running v.dissolve in a layer. In QGIS2 work the same script...

Processing algorithm…
Algorithm 'v.dissolve' starting…
Input parameters:
{ 'GRASS_MIN_AREA_PARAMETER' : 0.0001, 'GRASS_OUTPUT_TYPE_PARAMETER' : 0, 'GRASS_REGION_PARAMETER' : None, 'GRASS_SNAP_TOLERANCE_PARAMETER' : -1, 'GRASS_VECTOR_DSCO' : '', 'GRASS_VECTOR_LCO' : '', 'column' : 'FORMATIE', 'input' : '/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/gestuwd.shp', 'output' : '/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/25849cea13f848f3804011c7ba051148/output.gpkg' }

Attempting to use v.external for direct layer read
g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"
v.external input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/gestuwd.shp" output="vector_5c0e85082c4be11" --overwrite -o
g.region n=500000.0 s=471782.28125 e=266037.78125 w=231537.2496543302
v.dissolve input=vector_5c0e85082c4be11 column=FORMATIE output=output2ebe2052cd0c40dcb7ad19ca23723de2 --overwrite
v.out.ogr -c type="auto" input="output2ebe2052cd0c40dcb7ad19ca23723de2" output="/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/25849cea13f848f3804011c7ba051148/output.gpkg" format="GPKG" --overwrite
Starting GRASS GIS...
Executing </private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/grassdata/grass_batch_job.sh> ...
Default region was updated to the new projection, but if you have multiple mapsets `g.region -d` should be run in each to update the region from the default
Projection information updated
Over-riding projection check
Building topology for vector map <vector_5c0e85082c4be11@PERMANENT>...
Using external data format 'ESRI Shapefile' (feature type 'polygon')
Registering primitives...
262 primitives registered
18997 vertices registered
Number of nodes: 120
Number of primitives: 262
Number of points: 0
Number of lines: 0
Number of boundaries: 141
Number of centroids: 121
Number of areas: 141
Number of isles: 141
v.external complete. Link to vector map <vector_5c0e85082c4be11> created.
Traceback (most recent call last):
File "/usr/local/Cellar/grass7/7.4.3_2/grass-7.4.3/scripts/v.dissolve", line 110, in <module>
options, flags = grass.parser()
File "/usr/local/Cellar/grass7/7.4.3_2/grass-7.4.3/etc/python/grass/script/core.py", line 798, in parser
return _parse_opts(lines[1:])
File "/usr/local/Cellar/grass7/7.4.3_2/grass-7.4.3/etc/python/grass/script/core.py", line 749, in _parse_opts
os.environ[var] = val
File "/usr/local/opt/qgis3/libexec/vendor/bin/../lib/python3.7/os.py", line 682, in __setitem__
key = self.encodekey(key)
File "/usr/local/opt/qgis3/libexec/vendor/bin/../lib/python3.7/os.py", line 753, in encode
raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not bytes
ERROR: Vector map <output2ebe2052cd0c40dcb7ad19ca23723de2> not found
Execution of </private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/grassdata/grass_batch_job.sh> finished.
Cleaning up temporary files...
Starting GRASS GIS...
Executing </private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/grassdata/grass_batch_job.sh> ...
ERROR: Vector map <output2ebe2052cd0c40dcb7ad19ca23723de2> not found
Execution of </private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/grassdata/grass_batch_job.sh> finished.
Cleaning up temporary files...
Execution completed in 1.85 seconds
Results:
{'output': <QgsProcessingOutputLayerDefinition {'sink':/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/25849cea13f848f3804011c7ba051148/output.gpkg, 'createOptions': {'fileEncoding': 'System'}}>}

Loading resulting layers
The following layers were not correctly generated.<ul><li>/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/25849cea13f848f3804011c7ba051148/output.gpkg</li></ul>You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.

This time seems something related with python (again).

[luispuerto]

by the way... I just tried the vector buffer from gdal in QGIS2 and it's not working, but in QGIS3 it's...

I just testing.... but I really convinced that we are going to need to have two qgis for quite some time.

[fjperini]

@luispuerto You have that error:

TypeError: str expected, not bytes
ERROR: Vector map <output2ebe2052cd0c40dcb7ad19ca23723de2> not found`

I'll check.

TypeError: str expected, not bytes I solved that with:

os.environ['MANPATH'] = str(sys_man_path)

There may be another line with os.environ to change.

Verify that the change is applied in /grass7/7.4.3/libexec/bin/grass74

[luispuerto]

my set.paths function:

def set_paths(grass_config_dir):
    # addons (path)
    addon_path = os.getenv('GRASS_ADDON_PATH')
    if addon_path:
        for path in addon_path.split(os.pathsep):
            path_prepend(addon_path, 'PATH')

    # addons (base)
    addon_base = os.getenv('GRASS_ADDON_BASE')
    if not addon_base:
        addon_base = os.path.join(grass_config_dir, 'addons')
        os.environ['GRASS_ADDON_BASE'] = addon_base
    if not windows:
        path_prepend(os.path.join(addon_base, 'scripts'), 'PATH')
    path_prepend(os.path.join(addon_base, 'bin'), 'PATH')

    # standard installation
    if not windows:
        path_prepend(gpath('scripts'), 'PATH')
    path_prepend(gpath('bin'), 'PATH')

    # Set PYTHONPATH to find GRASS Python modules
    if os.path.exists(gpath('etc', 'python')):
        path_prepend(gpath('etc', 'python'), 'PYTHONPATH')

    # set path for the GRASS man pages
    grass_man_path = gpath('docs', 'man')
    addons_man_path = os.path.join(addon_base, 'docs', 'man')
    man_path = os.getenv('MANPATH')
    sys_man_path = None
    if man_path:
        path_prepend(addons_man_path, 'MANPATH')
        path_prepend(grass_man_path, 'MANPATH')
    else:
        try:
            nul = open(os.devnull, 'w')
            p = Popen(['manpath'], stdout=subprocess.PIPE, stderr=nul)
            nul.close()
            s = p.stdout.read()
            p.wait()
            sys_man_path = s.strip()
        except:
            pass

        if sys_man_path:
            # to_text_string disabled, see https://trac.osgeo.org/grass/ticket/3508
            # os.environ['MANPATH'] = to_text_string(sys_man_path)
            os.environ['MANPATH'] = str(sys_man_path)
            path_prepend(addons_man_path, 'MANPATH')
            path_prepend(grass_man_path, 'MANPATH')
        else:
            # to_text_string disabled, see https://trac.osgeo.org/grass/ticket/3508
            # os.environ['MANPATH'] = to_text_string(addons_man_path)
            os.environ['MANPATH'] = addons_man_path
            path_prepend(grass_man_path, 'MANPATH')

    # Set LD_LIBRARY_PATH (etc) to find GRASS shared libraries
    # this works for subprocesses but won't affect the current process
    path_prepend(gpath("lib"), ld_library_path_var)

[fjperini]

@luispuerto I checked that with v.buffer everything works fine, doing v.dissolve the error TypeError exits again.

It seems that now we have to correct the file: /usr/local/Cellar/grass7/7.4.3_2/grass-7.4.3/etc/python/grass/script/core.py. Since it is the last (of GRASS) to run.

[fjperini]

@luispuerto I'm testing others v.* and they are working well.

[luispuerto]

@fjperini you are right... what an eye have! I swear that v.dissolve want the second script I tried after v.buffer. I've been trying to use some other scripts —from r.* or i.* toolboxes— but I don't have anything at hand right now to test them —that I can say the script it's working properly.

[fjperini]

The other one that I found is: v.to.lines

I will try to solve the problem in core.py if that is the correct file. Ha!

It would be necessary to report them to the developers.

They are upstream problems anyway.

Everything else seems to work well.

[luispuerto]

Great 🎈

But if it’s an upstream problem why do they work on QGIS2?

_{Sent with GitHawk}

[fjperini]

@luispuerto It may be by the python version:

File "/usr/local/Cellar/grass7/7.4.3_2/grass-7.4.3/etc/python/grass/script/core.py"
...
File "/usr/local/opt/qgis3/libexec/vendor/bin/../lib/python3.7/os.py"

QGIS2 uses Python 2.

If you see in the comment

# to_text_string disabled, see https://trac.osgeo.org/grass/ticket/3508
# os.environ['MANPATH'] = to_text_string(sys_man_path)
os.environ['MANPATH'] = str(sys_man_path)

for example for the first case: v.buffer.

There are things that change and may need to be adjusted from version to version.
We just have to locate that os.environ we have to apply str to solve v.dissolve and v.to.lines and done.

[fjperini]

@luispuerto A line within def parser(): must be changed.
I'm working to make them compatible with Python 3. I hope to find the line that needs to be changed. Ha!

def parser():
    """Interface to g.parser, intended to be run from the top-level, e.g.:

    ::

        if __name__ == "__main__":
            options, flags = grass.parser()
            main()

    Thereafter, the global variables "options" and "flags" will be
    dictionaries containing option/flag values, keyed by lower-case
    option/flag names. The values in "options" are strings, those in
    "flags" are Python booleans.

    Overview table of parser standard options:
    https://grass.osgeo.org/grass74/manuals/parser_standard_options.html
    """
    if not os.getenv("GISBASE"):
        print("You must be in GRASS GIS to run this program.", file=sys.stderr)
        sys.exit(1)

    cmdline = [basename(encode(sys.argv[0]))]
    cmdline += [b'"' + encode(arg) + b'"' for arg in sys.argv[1:]]
    environ[b'CMDLINE'] = b' '.join(cmdline)

Maybe the problem is in

environ[b'CMDLINE'] = b' '.join(cmdline)

Definitely related os.environ, Python 3 requires str.

Python 2 and Python 3 differences:

In Python 2:
  Bytes == Strings
  Unicodes != Strings 
In Python 3:
  Bytes != Strings
  Unicodes == Strings 

Apparently, only v.dissolve and v.to.lines the only ones that do not work, all the others work well.

https://trac.osgeo.org/grass/wiki/Python3Support
https://gis.stackexchange.com/questions/304245/grass-processing-not-working-on-qgis
https://stackoverflow.com/questions/39949587/can-python-os-environ-get-ever-return-a-non-string
http://osgeo-org.1560.x6.nabble.com/Python-3-porting-and-unicode-td5344215.html

[fjperini]

Here is a solution mentioned:

https://stackoverflow.com/questions/52269281/fix-import-error-on-using-environb-in-python

# 1. at the top, add an import
import pipes
# 2. remove the `from os import environb as environ` line altogether
# 3. in def parse(), use
cmdline = [basename(sys.argv[0])]
cmdline += (pipes.quote(a) for a in sys.argv[1:])
os.environ['CMDLINE'] = ' '.join(cmdline)

We are very close to solving it.
This must be reported to the GRASS developers.

[fjperini]

@luispuerto @nickrobison I've solved v.dissolve and v.to.lines 🎉

With all the changes applied, GRASS should be completely compatible with Python 3.

While we wait for version 7.8/8.0, which apparently (https://gis.stackexchange.com/a/304598) will be compatible with Python 3.

Files to which I had to apply changes:

/etc/python/grass/script/core.py

--- a/etc/python/grass/script/core.py
+++ b/etc/python/grass/script/core.py
@@ -25,9 +25,11 @@
 import subprocess
 import shutil
 import codecs
+import string
+import random
 import types as python_types
 
-from .utils import KeyValue, parse_key_val, basename, encode
+from .utils import KeyValue, parse_key_val, basename, encode, decode
 from grass.exceptions import ScriptError, CalledModuleError
 
 # i18N
@@ -38,13 +40,13 @@
     # python2
     import __builtin__
     from os import environ
+    __builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.ugettext
 except ImportError:
     # python3
     import builtins as __builtin__
     from os import environb as environ
     unicode = str
-__builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.lgettext
-
+    __builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.gettext
 
 # subprocess wrapper that uses shell on Windows
 
@@ -104,6 +106,21 @@
     except TypeError:
         pass
     return bytes(val)
+
+
+def _make_unicode(val, enc):
+    """Convert value to unicode with given encoding
+
+    :param val: value to be converted
+    :param enc: encoding to be used
+    """
+    if val is None or enc is None:
+        return val
+    else:
+        if enc == 'default':
+            return decode(val)
+        else:
+            return decode(val, encoding=enc)
 
 
 def get_commands():
@@ -301,7 +318,8 @@
             continue
         # convert string to bytes
         opt = encode(opt)
-        if val != None:
+        prog = encode(prog)
+        if val is not None:
             if opt.startswith(b'_'):
                 opt = opt[1:]
                 warning(_("To run the module <%s> add underscore at the end"
@@ -328,7 +346,8 @@
     else:
         # TODO: construction of the whole command is far from perfect
         args = make_command(*args, **kwargs)
-        raise CalledModuleError(module=None, code=repr(args),
+        code = ''.join([decode(each) for each in args])
+        raise CalledModuleError(module=None, code=code,
                                 returncode=returncode)
 
 def start_command(prog, flags=b"", overwrite=False, quiet=False,
@@ -360,6 +379,9 @@
 
     :return: Popen object
     """
+    if 'encoding' in kwargs.keys():
+        encoding = kwargs.pop('encoding')
+
     options = {}
     popts = {}
     for opt, val in kwargs.items():
@@ -379,7 +401,6 @@
         sys.stderr.flush()
     return Popen(args, **popts)
 
-
 def run_command(*args, **kwargs):
     """Execute a module synchronously
 
@@ -408,11 +429,18 @@
 
     :raises: ``CalledModuleError`` when module returns non-zero return code
     """
+    encoding = 'default'
+    if 'encoding' in kwargs:
+        encoding = kwargs['encoding']
+
     if _capture_stderr and 'stderr' not in kwargs.keys():
         kwargs['stderr'] = PIPE
     ps = start_command(*args, **kwargs)
     if _capture_stderr:
         stdout, stderr = ps.communicate()
+        if encoding is not None:
+            stdout = _make_unicode(stdout, encoding)
+            stderr = _make_unicode(stderr, encoding)
         returncode = ps.poll()
         if returncode:
             sys.stderr.write(stderr)
@@ -466,10 +494,17 @@
 
     :return: stdout
     """
+    encoding = 'default'
+    if 'encoding' in kwargs:
+        encoding = kwargs['encoding']
+
     if _capture_stderr and 'stderr' not in kwargs.keys():
         kwargs['stderr'] = PIPE
     process = pipe_command(*args, **kwargs)
     stdout, stderr = process.communicate()
+    if encoding is not None:
+        stdout = _make_unicode(stdout, encoding)
+        stderr = _make_unicode(stderr, encoding)
     returncode = process.poll()
     if _capture_stderr and returncode:
         sys.stderr.write(stderr)
@@ -539,12 +574,22 @@
 
     :raises: ``CalledModuleError`` when module returns non-zero return code
     """
+    encoding = 'default'
+    if 'encoding' in kwargs:
+        encoding = kwargs['encoding']
     # TODO: should we delete it from kwargs?
     stdin = kwargs['stdin']
+    if encoding is None or encoding == 'default':
+        stdin = encode(stdin)
+    else:
+        stdin = encode(stdin, encoding=encoding)
     if _capture_stderr and 'stderr' not in kwargs.keys():
         kwargs['stderr'] = PIPE
     process = feed_command(*args, **kwargs)
     unused, stderr = process.communicate(stdin)
+    if encoding is not None:
+        unused = _make_unicode(unused, encoding)
+        stderr = _make_unicode(stderr, encoding)
     returncode = process.poll()
     if _capture_stderr and returncode:
         sys.stderr.write(stderr)
@@ -738,14 +783,15 @@
             break
         try:
             [var, val] = line.split(b'=', 1)
+            [var, val] = [decode(var), decode(val)]
         except:
             raise SyntaxError("invalid output from g.parser: %s" % line)
 
-        if var.startswith(b'flag_'):
+        if var.startswith('flag_'):
             flags[var[5:]] = bool(int(val))
-        elif var.startswith(b'opt_'):
+        elif var.startswith('opt_'):
             options[var[4:]] = val
-        elif var in [b'GRASS_OVERWRITE', b'GRASS_VERBOSE']:
+        elif var in ['GRASS_OVERWRITE', 'GRASS_VERBOSE']:
             os.environ[var] = val
         else:
             raise SyntaxError("invalid output from g.parser: %s" % line)
@@ -768,7 +814,7 @@
     "flags" are Python booleans.
 
     Overview table of parser standard options:
-    https://grass.osgeo.org/grass74/manuals/parser_standard_options.html
+    https://grass.osgeo.org/grass77/manuals/parser_standard_options.html
     """
     if not os.getenv("GISBASE"):
         print("You must be in GRASS GIS to run this program.", file=sys.stderr)
@@ -820,6 +866,30 @@
     os.mkdir(tmp)
 
     return tmp
+
+
+def tempname(length, lowercase=False):
+    """Generate a GRASS and SQL compliant random name starting with tmp_
+    followed by a random part of length "length"
+
+    :param int length: length of the random part of the name to generate
+    :param bool lowercase: use only lowercase characters to generate name
+    :returns: String with a random name of length "length" starting with a letter
+    :rtype: str
+
+    :Example:
+
+    >>> tempname(12)
+    'tmp_MxMa1kAS13s9'
+    """
+
+    chars = string.ascii_lowercase + string.digits
+    if not lowercase:
+        chars += string.ascii_uppercase
+    random_part = ''.join(random.choice(chars) for _ in range(length))
+    randomname = 'tmp_' + random_part
+
+    return randomname
 
 
 def _compare_projection(dic):
@@ -1270,7 +1340,7 @@
 
     :return: directory of mapsets/elements
     """
-    if isinstance(type, python_types.StringTypes) or len(type) == 1:
+    if isinstance(type, str) or len(type) == 1:
         types = [type]
         store_types = False
     else:

and to the remaining files just change file by open

/grass-7.4.3/scripts/db.py
/grass-7.4.3/scripts/v.db.addtable
/grass-7.4.3/scripts/v.dissolve
/grass-7.4.3/scripts/v.lines

note: file() is not supported in Python 3. Using open()

I will apply the changes now!

p/d: If a similar error arises, you already know where to start. Now I want that beer 😄 Ha!

[fjperini]

The globe support is almost finished, the option to use the old versions (osgearth 2.7) of osgearth openscenegraph apparently builds well.

But I found many things to change for the new versions (3.6.3). For that I will upload new formulas that includes the support for Qt5 necessary to build some libraries, for example:

/usr/local/opt/osgearth-qt5/lib/libosgEarthQt5.dylib

needed by

/PlugIns/qgis/libglobeplugin.dylib

Apparently I just need to correct an error (after many) so that the build is completed.

../src/plugins/globe/featuresource/qgsglobefeaturesource.h:33:134: error: non-virtual member function marked 'override' hides virtual member function
    osgEarth::Features::FeatureCursor *createFeatureCursor( const osgEarth::Symbology::Query &query = osgEarth::Symbology::Query() ) override;
                                                                                                                                     ^
/usr/local/opt/osgearth-qt5/include/osgEarthFeatures/FeatureSource:149:32: note: hidden overloaded virtual function 'osgEarth::Features::FeatureSource::createFeatureCursor' declared here: different number of parameters (2 vs 1)
        virtual FeatureCursor* createFeatureCursor(const Symbology::Query& query, ProgressCallback* progress) =0;
                               ^

I hope to solve it. Ha!

[fjperini]

@nickrobison As there are several changes for Python 3, it might be best to create a grass7-dev formula for QGIS 3 and use it momentarily until GRASS 7.8 / 8.0 arrives.

@luispuerto I'll give you the formula with the changes and the tests in QGIS 2 and QGIS3.

If something comes up in QGIS 2, we'll have to use grass7-dev.

[luispuerto]

Tell me which branch in your repo or which pull request I should use.

[fjperini]

@luispuerto You can already try it!

[luispuerto]

I just tested it v.dissolve in QGIS3&2 and it works perfectly in both. 👍

Again, great work!

[fjperini]

Once @nickrobison merge the changes we close this. Ha! Thanks @luispuerto!