"Killed"

[extremecarver]

hgt file /home/contourlines/hgt/SRTM1v3.0/N25E003.hgt: 3601 x 3601 points, bbox: (3.00000, 25.00000, 4.00000, 26.00000)
Killed

not sure why this is happening - command was:
nice -n 19 pyhgtmap --earthexplorer-user=test--earthexplorer-password=Test--jobs=10 --polygon=/home/contourlines/bounds/africa.poly --step=10 --no-zero-contour --void-range-max=-420 --output-prefix=africa --line-cat=$detail --start-node-id=10000000 --start-way-id=10000000 --source=view1,srtm1,view3,srtm3 --max-nodes-per-way=230 --max-nodes-per-tile=0 --pbf --hgtdir=/home/contourlines/hgt -j16

With around 62GB of available RAM. Did id run out of memory? I will rerun with --jobs=2 but a bit better error message why it was killed would be great. Europe was killed as well. with above command.

[extremecarver]

Tried with jobs=2 and it got killed exactly at the same place after writing 1.9GB of data

[agrenott]

There was indeed a memory leak, keeping data between tiles.
It should be fixed now. I've been able to generate significant part of Africa (4.7GB output) before aborting.
Please note ways are accumulated until the end, and file is committed at once by pyosmium, so memory requirement for such an area is still significant.

[agrenott]

FYI, I've been able to finish Africa with a single output file. Took ~80 minutes (with the pending pyosmium optimization) and output is 6.3 GB.

[extremecarver]

Phyghtmap was using less memory compared to even after fixing this. With 64GB RAM I cannot compile Africa 10m interval (13.9GB) anymore. Not to mention Asia 10m (66GB).
I'm not sure if there is maybe another bug or the changes improve performance at the expense of memory. It aborted at nearly 11GB.

[agrenott]

Are you sure you're using the 3.1 version?
I'm able to generate the africa 10m, despite using a lot of memory:

But then I agree 24GB of memory seems excessive to output a file which is "only" 6.3GB... There's probably some useless accumulation of objects somewhere.

[extremecarver]

Africa 10m interval should be around 13.9GB in size (I'm using 1" source data exclusively, alos world 3d newest Revision, but I guess any 1" data will be similar for memory consumption).

6.3GB means something is wrong it that's for 10m interval so I guess that's 20m interval maybe with 3" data?

With the old phyghtmap it was possible (using 1 process) to generate Asia 10m resulting in a 66GB output file with 64GB RAM and I think a 6GB temp file on NVME (though I'm not sure that the temp was used, but it was really close with memory and calculated like 3.5 days nonstop on a AMD Ryzen 3600)

[extremecarver]

Actually right now with 10.6GB written out - it uses 61GB RAM already. Something is definitely wrong still or we will need to use the phyghtmap with the first patch to get it running for Asia/China/Russia/South America/Africa on a 64GB RAM machine (haven*t tried to increase virtual memory to say 128GB or similar - which maybe okay for the usecase). That is with jobs=9

[agrenott]

Well, I'm actually using the command you originally shared, so 10m steps. The difference being it's 1" and 3" (but 1" should be favored when available).

pyhgtmap --jobs=10 --polygon=africa.poly --step=10 --no-zero-contour --void-range-max=-420 --output-prefix=africa  --start-node-id=10000000 --start-way-id=10000000 --source=view1,srtm1,view3,srtm3 --max-nodes-per-way=230 --max-nodes-per-tile=0 --pbf --hgtdir=/mnt/g/git/garmin_mtb/work/hgt/ --log=DEBUG

[extremecarver]

Well there is no viewfinderpanoramas 1" - and the SRTM1" now require an account to download - which I guess you haven't done (not even sure it's implemented). Alos World 3D was even harder to download - I think I wrote a script based on their website links to download them - but yeah I know 1" take much longer to process and likely use more memory too. But I guess there is still some memory leak somewhere else It really should not hit 64GB of RAM with 10.8GB of data written out (and overall should be 13.9GB). Maybe pyhgtmap could use a temp file to store the ways with a switch?

[agrenott]

I've setup an account for SRTM1", and it works fine.
Anyway, the memory usage seems in fact logical, beside being un-optimized: I need to keep the list of ways to be written after all nodes. Each way is a tuple of 3 ints and a bool, but it actually uses A LOT of memory in Python:

> from pympler import asizeof
> a=pyhgtmap.output.WayType(next_node_id, len(contour), is_closed_way, elevation)
> asizeof.asized(a, detail=1).format()
'WayType(first_node_id=10000000, nb_nodes=1695, closed_loop=False, elevation=0) size=184 flat=72\n    10000000 size=32 flat=32\n    1695 size=32 flat=32\n    False size=24 flat=24\n    0 size=24 flat=24\n    __class__ size=0 flat=0'

184 bytes for each way. As the output for Africa has (tens of?) millions of ways, the actual memory usage is HUGE...
I'll try to see whether more efficient storage (array or numpy.array) improves this significantly.

[extremecarver]

But how was it possible with the old phyghtmap to use approx only 1/6 of memory? I think the easiest is just an option for a temp file. Yeah that would mean 360GB or so for Asia, but that is much easier to have on a computer than 360GB of memory... (and increasing swap for that size I'm not sure how good the memory management in Linux could handle that). If one is to compute such a huge file, some hundred GB of tempfile space shouldn't be an issue. Storing all those hgts anyhow takes up huge space too so someone trying this likely has a big HDD for data storage besides a fast system drive anyhow.

[extremecarver]

It's not that I actually need this - lately there have been no significant updates to 1" source data outside of Europe and north America when it comes to higher quality - and I still have the old pbf output files - I noticed it with Europe and then just wanted to give Africa a try being a couple of times bigger.

[agrenott]

My Africa dataset contains almost 100 million ways. Relying on numpy array instead of list of tuples reduces memory consumption by about 10x factor (2.6GB consumed at the end of the process):

time pyhgtmap --jobs=10 --polygon=africa.poly --step=10 --no-zero-contour --void-range-max=-420 --output-prefix=africa  --start-node-id=10000000 --start-way-id=10000000 --source=view1,srtm1,view3,srtm3 --max-nodes-per-way=230 --max-nodes-per-tile=0 --pbf --hgtdir=/mnt/g/git/garmin_mtb/work/hgt/ --log=DEBUG
...
2023-06-02 01:56:04,878 - pyhgtmap.output:79:DEBUG - 27675 - done() - Writing 92957050 pending ways
2023-06-02 02:10:53,604 - pyhgtmap.output:82:DEBUG - 27675 - done() - done!

real    50m31.503s
user    20m44.594s
sys     0m12.000s

[extremecarver]

Well I now have another problem with the newest version - seems there are problems with order - when trying to split the output with mkgmap splitter - I get the error:
Error: Node ids are not sorted. Use e.g. osmosis to sort the input data.
This is not supported with keep-complete=true or --problem-list
uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted

Phygtmap produced data never had that problem. Running osmosis on this data to sort it - would again fail due to size constraints.

I tried if it's related to a map being south of the Equator - and used Fiji which was fine except for getting this error message although I don't see a problem with the data output on first glance:

hgt file /home/contourlines/hgt/SRTM1v3.0/S20W180.hgt: 3601 x 3601 points, bbox: (-180.00000, -20.00000, -179.00000, -19.00000), checking polygon borders
/usr/local/lib/python3.10/dist-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)

Not sure how what this means. Maybe input data is problematic on that tile?

I then tried Ecuador to have a map passing the equator. Also no problem.
France neither. Seems to affect only very big countries - and I think that problem already has been there with pyhgtmap v 3.1 (but I don't know right now - just know that splitting Europe with mkgmap splitter failed which was produced by v 3.1

[agrenott]

That's a bit strange as I didn't change the nodes management with the latest change.
Could the error message reference the nodes references inside a way?
Or if it's only on big dataset, it might have "always" been here, possibly due to some integer overflow on nodes IDs... Any way to get more details by increasing verbosity of osmosis? (seeing the ID going from big positive to negative value could confirm the hypothesis).

[extremecarver]

Those files crash Osmosis running out of memory If I try to use Osmosis to sort it.
I could only try to run the patched phyghtmap on Europa and see if it goes through - I guess yes. Especially with Africa I don't think any of my input files changed since last succesful run.

The error message is from mkgmap splitter.

[extremecarver]

BTW - I downloaded a fresh copy of S20W180 from SRTM and the above small bug still happens. (this source gives directly hgt not tif files for download)
https://dwtkns.com/srtm30m/

Also the download tool crashed when it tried to download with correct username/password (or maybe crashed on unzipping):
S20W180: trying view1 ...
S20W180: trying srtm1v3.0 ...
S20W180: downloading file https://earthexplorer.usgs.gov/download/5e83a3efe0103743/SRTM1S20W180V3/EE to /home/contourlines/hgt/SRTM1v3.0/S20W180.tif ...
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 779, in downloadAndUnzip_Tif
os.stat(saveFilename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/contourlines/hgt/SRTM1v3.0/S20W180.tif'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/pyhgtmap", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/main.py", line 561, in main
hgtDataFiles = NASASRTMUtil.getFiles(
File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 887, in getFiles
saveFilename = getFile(area, source)
File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 876, in getFile
return downloadAndUnzip(url, area, source)
File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 762, in downloadAndUnzip
return downloadAndUnzip_Tif(url, area, source)
File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 784, in downloadAndUnzip_Tif
downloadToFile(url, saveFilename, source)
File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 756, in downloadToFile
return downloadToFile_SRTMv3(url, filename)
File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 731, in downloadToFile_SRTMv3
opener = earthexplorerLogin()
File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 726, in earthexplorerLogin
res2 = opener.open(req2)
File "/usr/lib/python3.10/urllib/request.py", line 525, in open
response = meth(req, response)
File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
response = self.parent.error(
File "/usr/lib/python3.10/urllib/request.py", line 563, in error
return self._call_chain(*args)
File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized

[extremecarver]

I will try tomorrow using this version (I think this one must still be close to identical and I believe it will run under current Ubuntu):
https://github.com/agrenott/pyhgtmap/tree/798b96552cfae00a7db764c5e26d790b20feedb2

or should I even downgrade to this one:
https://github.com/agrenott/pyhgtmap/tree/6062a7b75fcf7e70d63843daad9d624f3280e279
?

At the same time I'm just running the most recent version vs South-America and Asia and see if it splits or not. On both continents I definitely didn't change any input data since the last succesful run.

[agrenott]

BTW - I downloaded a fresh copy of S20W180 from SRTM and the above small bug still happens. (this source gives directly hgt not tif files for download) https://dwtkns.com/srtm30m/

Also the download tool crashed when it tried to download with correct username/password (or maybe crashed on unzipping): S20W180: trying view1 ... S20W180: trying srtm1v3.0 ... S20W180: downloading file https://earthexplorer.usgs.gov/download/5e83a3efe0103743/SRTM1S20W180V3/EE to /home/contourlines/hgt/SRTM1v3.0/S20W180.tif ... Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 779, in downloadAndUnzip_Tif os.stat(saveFilename) FileNotFoundError: [Errno 2] No such file or directory: '/home/contourlines/hgt/SRTM1v3.0/S20W180.tif'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/bin/pyhgtmap", line 8, in sys.exit(main()) File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/main.py", line 561, in main hgtDataFiles = NASASRTMUtil.getFiles( File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 887, in getFiles saveFilename = getFile(area, source) File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 876, in getFile return downloadAndUnzip(url, area, source) File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 762, in downloadAndUnzip return downloadAndUnzip_Tif(url, area, source) File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 784, in downloadAndUnzip_Tif downloadToFile(url, saveFilename, source) File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 756, in downloadToFile return downloadToFile_SRTMv3(url, filename) File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 731, in downloadToFile_SRTMv3 opener = earthexplorerLogin() File "/usr/local/lib/python3.10/dist-packages/pyhgtmap/NASASRTMUtil.py", line 726, in earthexplorerLogin res2 = opener.open(req2) File "/usr/lib/python3.10/urllib/request.py", line 525, in open response = meth(req, response) File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response response = self.parent.error( File "/usr/lib/python3.10/urllib/request.py", line 563, in error return self._call_chain(*args) File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain result = func(*args) File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 401: Unauthorized

urllib.error.HTTPError: HTTP Error 401: Unauthorized

You have a credentials issue. You may delete your ~/.pyhgtmaprc to ensure it's not trying to use an old one.