Geocode creates temp files but never completes it's process

[TheJeran]

• which operating system are you using?
  Linux
• which environment is pyroSAR running in?
  Anaconda Docker env
• which version of pyroSAR are you using?
  '0.15.1' from conda
• which function of pyroSAR did you call with which parameters?
  geocode(infile=check, outdir='test2')
• if applicable, which version of SNAP or GAMMA are you using in pyroSAR?

  snap version
/opt/snap/bin/../platform/lib/nbexec: WARNING: environment variable DISPLAY is not set
There are parameters but nobody wants to proces them: [version]

• the full error message
  none it just never ends. I think it's a SNAP issue actually but have not the knowledge to figure out what exactly.

[johntruckenbrodt]

Hi @TheJeran thanks for reaching out. Can you please run geocode with test=True:

geocode(infile=check, outdir='test2', test=True)

This will only create the workflow but not execute it.
Then you can run the workflow directly over the command line:

gpt workflow.xml

This way you can get all the SNAP command line info and perhaps some insight into what is going on.
One idea.. are you running this behind a proxy? Then SNAP is not able to download the needed DEM and OSV files. pyroSAR can download it for you, see here and here.

[TheJeran]

Thanks for the response John.

I ran that code and it exported a S1B__IW___A_20190403T020824_Cal_NR_Orb_ML_TF_TC_dB_proc.xml file. I don't know if this is the expected behavior or if it was supposed to be 'workflow.xml'

My docker doesn't have a gpt funtion so
gpt S1B__IW___A_20190403T020824_Cal_NR_Orb_ML_TF_TC_dB_proc.xml

bash: gpt: command not found

is the outcome.

This lack of gpt functionality could be the cause?

[johntruckenbrodt]

No no, workflow.xml was just a placeholder, your workflow name looks good.
pyroSAR needs to be able to use gpt. If this is not available then you cannot do any processing.
This is surprising though. pyroSAR is supposed to raise an error when gpt cannot be found. What does the following return?

from pyroSAR.examine import ExamineSnap
print(ExamineSnap().gpt)

[TheJeran]

/opt/snap/bin/gpt

[johntruckenbrodt]

Okay so it found the gpt executable. Try running this in your shell:

/opt/snap/bin/gpt S1B__IW___A_20190403T020824_Cal_NR_Orb_ML_TF_TC_dB_proc.xml

Or this:

export PATH=$PATH:/opt/snap/bin
gpt S1B__IW___A_20190403T020824_Cal_NR_Orb_ML_TF_TC_dB_proc.xml

[TheJeran]

Wonderful John! So it does seem to be a connection error and the connections time out. I checked the URLs in my browser and they throw me a 404.
http://step.esa.int/auxdata/dem/SRTMGL1/N44W126.SRTMGL1.hgt.zip

Any ideas? This is a SNAP thing right?

[johntruckenbrodt]

This particular tile does not exist because this is an ocean area. The westernmost tile on that latitude is N44W125.SRTMGL1.hgt.zip.
SNAP should not be complaining here. Try leaving out the terrain flattening step. This is for land areas only.

geocode(infile=check, outdir='test2', terrainFlattening=False)

[TheJeran]

It's still trying to download the same files. :/

geocode(infile=name, outdir='test2', terrainFlattening=False, test=True)

....INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N44W125.SRTMGL1.hgt.zip INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N44W123.SRTMGL1.hgt.zip INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N44W124.SRTMGL1.hgt.zip INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N44W126.SRTMGL1.hgt.zip INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N44W122.SRTMGL1.hgt.zip

[johntruckenbrodt]

Actually, just do this:

geocode(infile=check, outdir='test2', terrainFlattening=False, demName='GETASSE30')

This GETASSE30 DEM is completely global, i.e., also has tiles over ocean. It's resolution is really low but this doesn't matter in your case.

[TheJeran]

Still trying to download them :/
It does more without test=True but still hangs. So I dunno

I have no idea which files are near the ocean. Sentinel-1 is not user-friendly.

[johntruckenbrodt]

I just tried myself with this scene. Apologies, it is of course mostly over land and terrain flattening is a good idea.
It works fine for me with both GETASSE30 and SRTM 1Sec HGT DEMs:

>gpt S1B__IW___A_20190403T020824_Cal_NR_Orb_ML_TC_dB_proc.xml
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL not found on system. Internal GDAL 3.0.0 from distribution will be used. (f1)
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
version = 2.91
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N44W125.SRTMGL1.hgt.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N44W126.SRTMGL1.hgt.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://step.esa.int/auxdata/dem/SRTMGL1/N44W126.SRTMGL1.hgt.zip on http://step.esa.int/auxdata/dem/SRTMGL1/N44W126.SRTMGL1.hgt.zip
.INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N44W124.SRTMGL1.hgt.zip
.INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N44W123.SRTMGL1.hgt.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N44W122.SRTMGL1.hgt.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N43W126.SRTMGL1.hgt.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://step.esa.int/auxdata/dem/SRTMGL1/N43W126.SRTMGL1.hgt.zip on http://step.esa.int/auxdata/dem/SRTMGL1/N43W126.SRTMGL1.hgt.zip
.INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N43W125.SRTMGL1.hgt.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N43W124.SRTMGL1.hgt.zip
.INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N43W123.SRTMGL1.hgt.zip
10%INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N43W122.SRTMGL1.hgt.zip
....20%....30%....40%....50%..INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N42W126.SRTMGL1.hgt.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://step.esa.int/auxdata/dem/SRTMGL1/N42W126.SRTMGL1.hgt.zip on http://step.esa.int/auxdata/dem/SRTMGL1/N42W126.SRTMGL1.hgt.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N42W125.SRTMGL1.hgt.zip
.INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N42W124.SRTMGL1.hgt.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N42W123.SRTMGL1.hgt.zip
.INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N42W122.SRTMGL1.hgt.zip
60%....70%....80%....90% done.

If you are behind a proxy and SNAP cannot download those files then I don't have a workaround. However, the times I have observed SNAP having problems with this it did at least write a message saying it is not able to download. Are you sure the process hangs or is it just taking a long time to do anything (because of limited hardware resources)?

[TheJeran]

I think it may be a proxy issue since I can access those URLs through the browser. I'll look into it. Do you know where these files get stored to? if in the worst case scenario I can download them manually.

[johntruckenbrodt]

Some of the tiles you cannot download manually because they don't exist. There are no tiles over ocean. You can however create a custom DEM mosaic and pass that to geocode as externalDEMFile.

See here for some more details about DEM handling in pyroSAR: https://pyrosar.readthedocs.io/en/latest/general/DEM.html

[TheJeran]

I realize this is a different issue John. But I'm doin a local test on my Windows laptop to make sure its a proxy issue, but I can't seem to get pyroSAR to recognize my SNAP install in the PATH variable. Any idea how to solve this error for windows?

SNAP could not be identified. If you have installed it please add the path to the SNAP executables (bin subdirectory) to the PATH environment. E.g. in the Linux .bashrc file add the following line: export PATH=$PATH:path/to/snap/bin"

It's confusing since it can wget the files but not downloading them via gpt.

[johntruckenbrodt]

Like it says in the message. Add the SNAP directory to the PATH environment variable.

[TheJeran]

I probably should have included I have already done that. I tried the exe's directly too.

Does it use 32 bit or something?

[johntruckenbrodt]

the PATH variable is for paths only, not executables.

[TheJeran]

I figured. But that didn't seem to work, since the error message mentioned the executables I tried those as well to no avail. Does pyroSAR require a specific version? should snap64 work?

[johntruckenbrodt]

Okay that's strange. The method pyroSAR.examine.ExamineSnap.__identify_snap checks whether the following exist:

1. An executable called snap, snap.exe, snap32.exe or snap64.exe in one of the directories listed in PATH
2. A directory etc next to the executable's bin directory
3. A file snap.auxdata.properties in this etc directory
4. An executable gpt or gpt.exe in the same directory as the snap* executable

Can you please check whether all these conditions are met?

[TheJeran]

The love of programming eh?
I've got a meeting with Felix tomorrow to hopefully fix the cluster access issues. In the mean time let me blow your mind with this:

2. See 3

4. See 1

[johntruckenbrodt]

Oh yes, a never dying love. Do I know this Felix?
This looks all good. Can you check whether this directory is actually in your path environment during runtime:

import os
for path in os.environ.get('PATH').split(os.pathsep):
    print(path)

Also, please have a look at the file ~/.pyrosar/config.ini in your user directory. If there is something weird written in there it is worth deleting it so that pyroSAR will create a new one.

[TheJeran]

My understanding was that he helped develop pyroSAR (he's listed as a contributor on the github)

Ahhh... so that could be the issue

Do you know how to enter a path to this list?

[johntruckenbrodt]

Ah of course, this Felix. I just didn't see the connection between him and you. Where do you know each other from?
These variables are read only once upon start of an IDE/terminal/etc. Try restarting whatever you are using to run pyroSAR and check again. If that doesn't help try restarting your computer.

[TheJeran]

sigh if only it were that simple. Still the same after restart.

Felix is at my institute and was the one who suggested pyroSAR, which should solve my sentinel-1 problems whenever iI can get it to function. I know another project I worked on earlier is probably going to start using Sentinel-1 data in the near future so would be good to get this working somewhere. I'll see what our session today yields.

[johntruckenbrodt]

Okay so that is beyond the scope of pyroSAR then. You will have to find a way of how to get the path C:\Program Files\snap\bin into the PATH variable so that you can display it via os.environ.get('PATH'). Once this works pyroSAR should be able to find the SNAP installation.

[johntruckenbrodt]

I am closing this because there hasn't been an update in several months. Feel free to reopen any time.