Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't build pbfparser #199

Closed
YaroShkvorets opened this issue Oct 3, 2019 · 13 comments
Closed

Can't build pbfparser #199

YaroShkvorets opened this issue Oct 3, 2019 · 13 comments

Comments

@YaroShkvorets
Copy link

On Windows with JDK 8, following mapcreation.md guide I can successfully build brouter.jar but when I try to build pbfparser with

javac -d . -cp "brouter.jar;protobuf.jar;osmosis.jar" *.java

I get this error:

OsmParser2.java:15: error: cannot find symbol
public class OsmParser2 extends MapCreatorBase
                                ^
  symbol: class MapCreatorBase
OsmParser2.java:19: error: cannot find symbol
  private NodeListener nListener;
          ^
  symbol:   class NodeListener
  location: class OsmParser2
OsmParser2.java:20: error: cannot find symbol
  private WayListener wListener;
          ^
  symbol:   class WayListener
  location: class OsmParser2
OsmParser2.java:21: error: cannot find symbol
  private RelationListener rListener;
          ^
  symbol:   class RelationListener
  location: class OsmParser2
OsmParser2.java:24: error: cannot find symbol
                        NodeListener nListener,
                        ^
  symbol:   class NodeListener
  location: class OsmParser2
OsmParser2.java:25: error: cannot find symbol
                        WayListener wListener,
                        ^
  symbol:   class WayListener
  location: class OsmParser2
OsmParser2.java:26: error: cannot find symbol
                        RelationListener rListener ) throws Exception
                        ^
  symbol:   class RelationListener
  location: class OsmParser2
OsmParser2.java:69: error: cannot find symbol
    NodeData n = new NodeData( nid, lon, lat );
    ^
  symbol:   class NodeData
  location: class OsmParser2
OsmParser2.java:69: error: cannot find symbol
    NodeData n = new NodeData( nid, lon, lat );
                     ^
  symbol:   class NodeData
  location: class OsmParser2
OsmParser2.java:83: error: cannot find symbol
    WayData w = new WayData( wid, nodes );
    ^
  symbol:   class WayData
  location: class OsmParser2
OsmParser2.java:83: error: cannot find symbol
    WayData w = new WayData( wid, nodes );
                    ^
  symbol:   class WayData
  location: class OsmParser2
OsmParser2.java:98: error: cannot find symbol
    RelationData r = new RelationData( rid, wayIds, roles );
    ^
  symbol:   class RelationData
  location: class OsmParser2
OsmParser2.java:98: error: cannot find symbol
    RelationData r = new RelationData( rid, wayIds, roles );
                         ^
  symbol:   class RelationData
  location: class OsmParser2
13 errors

What could be the problem?
Sorry if it's a dumb question, Java is new for me.

Here's contents of brouter.jar:

META-INF/
META-INF/MANIFEST.MF
btools/
btools/router/
btools/router/KinematicModel.class
btools/router/KinematicPath.class
btools/router/KinematicPrePath.class
btools/router/MessageData.class
btools/router/OsmNodeNamed.class
btools/router/OsmNogoPolygon$Point.class
btools/router/OsmNogoPolygon.class
btools/router/OsmPath.class
btools/router/OsmPathElement.class
btools/router/OsmPathElementWithTraffic.class
btools/router/OsmPathModel.class
btools/router/OsmPrePath.class
btools/router/OsmTrack$1.class
btools/router/OsmTrack$OsmPathElementHolder.class
btools/router/OsmTrack.class
btools/router/ProfileCache.class
btools/router/RoutingContext.class
btools/router/RoutingEngine.class
btools/router/RoutingHelper.class
btools/router/RoutingIslandException.class
btools/router/SearchBoundary.class
btools/router/StdModel.class
btools/router/StdPath.class
btools/router/VoiceHint.class
btools/router/VoiceHintList.class
btools/router/VoiceHintProcessor.class
META-INF/maven/
META-INF/maven/org.btools/
META-INF/maven/org.btools/brouter-core/
META-INF/maven/org.btools/brouter-core/pom.xml
META-INF/maven/org.btools/brouter-core/pom.properties
btools/util/
btools/util/BitCoderContext.class
btools/util/ByteArrayUnifier.class
btools/util/ByteDataReader.class
btools/util/ByteDataWriter.class
btools/util/CheapAngleMeter.class
btools/util/CheapRuler.class
btools/util/CompactLongMap.class
btools/util/CompactLongSet.class
btools/util/Crc32.class
btools/util/DenseLongMap.class
btools/util/DiffCoderDataInputStream.class
btools/util/DiffCoderDataOutputStream.class
btools/util/FastMath.class
btools/util/FrozenLongMap.class
btools/util/FrozenLongSet.class
btools/util/IByteArrayUnifier.class
btools/util/LazyArrayOfLists.class
btools/util/LongList.class
btools/util/LruMap.class
btools/util/LruMapNode.class
btools/util/MixCoderDataInputStream.class
btools/util/MixCoderDataOutputStream.class
btools/util/ReducedMedianFilter.class
btools/util/SortedHeap$SortedBin.class
btools/util/SortedHeap.class
btools/util/StackSampler.class
btools/util/TinyDenseLongMap.class
META-INF/maven/org.btools/brouter-util/
META-INF/maven/org.btools/brouter-util/pom.xml
META-INF/maven/org.btools/brouter-util/pom.properties
btools/codec/
btools/codec/DataBuffers.class
btools/codec/IntegerFifo3Pass.class
btools/codec/LinkedListContainer.class
btools/codec/MicroCache.class
btools/codec/MicroCache2.class
btools/codec/NoisyDiffCoder.class
btools/codec/StatCoderContext.class
btools/codec/TagValueCoder$TagValueSet$FrequencyComparator.class
btools/codec/TagValueCoder$TagValueSet.class
btools/codec/TagValueCoder$TreeNode.class
btools/codec/TagValueCoder.class
btools/codec/TagValueValidator.class
btools/codec/TagValueWrapper.class
btools/codec/WaypointMatcher.class
META-INF/maven/org.btools/brouter-codec/
META-INF/maven/org.btools/brouter-codec/pom.xml
META-INF/maven/org.btools/brouter-codec/pom.properties
btools/mapaccess/
btools/mapaccess/DirectWeaver.class
btools/mapaccess/GeometryDecoder.class
btools/mapaccess/MatchedWaypoint.class
btools/mapaccess/NodesCache.class
btools/mapaccess/NodesList.class
btools/mapaccess/OsmFile.class
btools/mapaccess/OsmLink.class
btools/mapaccess/OsmLinkHolder.class
btools/mapaccess/OsmNode.class
btools/mapaccess/OsmNodePairSet$1.class
btools/mapaccess/OsmNodePairSet$OsmNodePair.class
btools/mapaccess/OsmNodePairSet.class
btools/mapaccess/OsmNodesMap.class
btools/mapaccess/OsmPos.class
btools/mapaccess/OsmTransferNode.class
btools/mapaccess/PhysicalFile.class
btools/mapaccess/Rd5DiffTool.class
btools/mapaccess/StorageConfigHelper.class
btools/mapaccess/TurnRestriction.class
btools/mapaccess/WaypointMatcherImpl.class
META-INF/maven/org.btools/brouter-mapaccess/
META-INF/maven/org.btools/brouter-mapaccess/pom.xml
META-INF/maven/org.btools/brouter-mapaccess/pom.properties
btools/expressions/
btools/expressions/BExpression.class
btools/expressions/BExpressionContext.class
btools/expressions/BExpressionContextNode.class
btools/expressions/BExpressionContextWay.class
btools/expressions/BExpressionLookupValue.class
btools/expressions/BExpressionMetaData.class
btools/expressions/CacheNode.class
btools/expressions/ProfileComparator.class
btools/expressions/VarWrapper.class
META-INF/maven/org.btools/brouter-expressions/
META-INF/maven/org.btools/brouter-expressions/pom.xml
META-INF/maven/org.btools/brouter-expressions/pom.properties
btools/mapcreator/
btools/mapcreator/ConvertSrtmTile.class
btools/mapcreator/ConvertUrlList.class
btools/mapcreator/DPFilter.class
btools/mapcreator/MapCreatorBase.class
btools/mapcreator/NodeCutter.class
btools/mapcreator/NodeData.class
btools/mapcreator/NodeFilter.class
btools/mapcreator/NodeIterator.class
btools/mapcreator/NodeListener.class
btools/mapcreator/OsmCutter.class
btools/mapcreator/OsmFastCutter.class
btools/mapcreator/OsmLinkP.class
btools/mapcreator/OsmNodeP.class
btools/mapcreator/OsmNodePT.class
btools/mapcreator/OsmParser.class
btools/mapcreator/OsmTrafficMap$OsmTrafficElement.class
btools/mapcreator/OsmTrafficMap.class
btools/mapcreator/PosUnifier.class
btools/mapcreator/RasterCoder.class
btools/mapcreator/RelationData.class
btools/mapcreator/RelationListener.class
btools/mapcreator/RelationMerger.class
btools/mapcreator/RelationStatistics.class
btools/mapcreator/RestrictionCutter.class
btools/mapcreator/RestrictionCutter5.class
btools/mapcreator/RestrictionData.class
btools/mapcreator/SrtmData.class
btools/mapcreator/SrtmRaster$Weights.class
btools/mapcreator/SrtmRaster.class
btools/mapcreator/WayCutter.class
btools/mapcreator/WayCutter5.class
btools/mapcreator/WayData.class
btools/mapcreator/WayIterator.class
btools/mapcreator/WayLinker$ThreadController.class
btools/mapcreator/WayLinker.class
btools/mapcreator/WayListener.class
META-INF/maven/org.btools/brouter-map-creator/
META-INF/maven/org.btools/brouter-map-creator/pom.xml
META-INF/maven/org.btools/brouter-map-creator/pom.properties
btools/server/
btools/server/request/
btools/server/Area.class
btools/server/BRouter.class
btools/server/CgiUpload.class
btools/server/NearRecentWps.class
btools/server/Polygon.class
btools/server/request/ProfileUploadHandler.class
btools/server/request/RequestHandler.class
btools/server/request/ServerHandler.class
btools/server/RouteServer.class
btools/server/ServiceContext.class
btools/server/SuspectManager$SuspectList.class
btools/server/SuspectManager.class
META-INF/maven/org.btools/brouter-server/
META-INF/maven/org.btools/brouter-server/pom.xml
META-INF/maven/org.btools/brouter-server/pom.properties```
@YaroShkvorets
Copy link
Author

After adding
import btools.mapcreator.*;
line to OsmParser2.java looks better. The following error remains though:

OsmParser2.java:99: error: no suitable constructor found for RelationData(long,LongList,List<String>)
    RelationData r = new RelationData( rid, wayIds, roles );
                     ^
    constructor RelationData.RelationData(long) is not applicable
      (actual and formal argument lists differ in length)
    constructor RelationData.RelationData(long,LongList) is not applicable
      (actual and formal argument lists differ in length)
1 error

@abrensch
Copy link
Owner

abrensch commented Oct 3, 2019

"OsmParser2" shouldn't be there, it was for something experimental which is no in the experimantal branch only.

So I deleted it.

@abrensch abrensch closed this as completed Oct 3, 2019
@hb0
Copy link

hb0 commented Mar 17, 2020

I hope it's ok to revive this issue - the following error occurs using javac -d . -cp "brouter.jar:protobuf.jar:osmosis.jar" *.java on the latest master:

user@HBWS:~/workspace/brouter_segments/pbfparser$ javac -d . -cp "brouter.jar:protobuf.jar:osmosis.jar" *.java
BPbfBlobDecoder2.java:27: error: cannot find symbol
    private OsmParser2 parser;
            ^
  symbol:   class OsmParser2
  location: class BPbfBlobDecoder2
BPbfBlobDecoder2.java:36: error: cannot find symbol
    public BPbfBlobDecoder2( String blobType, byte[] rawBlob, OsmParser2 parser )
                                                              ^
  symbol:   class OsmParser2
  location: class BPbfBlobDecoder2
2 errors

** Update 1 **
Forget it, my bad.

** Update 2 **
It seems to work without the BPbfBlobDecorder2.java file:
javac -d . -cp "brouter.jar:protobuf.jar:osmosis.jar" BPbfBlobDecoder.java BPbfFieldDecoder.java OsmParser.java

I stopped here, to be continued:

****************************
*** WayCutter5: Soft-Cut way-data into tiles
*** NodeCutter: Cut big node-tiles into 5x5 tiles
*** PosUnifier: Unify position values and enhance elevation
*** WayLinker: Format a region of an OSM map for routing
BRouter 1.6.1 / 01032020
parsed profile /home/user/workspace/brouter_original/misc/profiles2/car-traffic_analysis.brf cached=false
Error (linksProcessed=0 open paths: 0): datafile E5_N45.rd5 not found
datafile E5_N45.rd5 not found
BRouter 1.6.1 / 01032020
parsed profile /home/user/workspace/brouter_original/misc/profiles2/car-traffic_analysis.brf cached=false
Error (linksProcessed=0 open paths: 0): datafile E5_N50.rd5 not found
datafile E5_N50.rd5 not found
BRouter 1.6.1 / 01032020
parsed profile /home/user/workspace/brouter_original/misc/profiles2/car-traffic_analysis.brf cached=false
Error (linksProcessed=0 open paths: 0): datafile E10_N50.rd5 not found
datafile E10_N50.rd5 not found
BRouter 1.6.1 / 01032020
parsed profile /home/user/workspace/brouter_original/misc/profiles2/car-traffic_analysis.brf cached=false
Error (linksProcessed=0 open paths: 0): datafile E10_N45.rd5 not found
datafile E10_N45.rd5 not found
BRouter 1.6.1 / 01032020
parsed profile /home/user/workspace/brouter_original/misc/profiles2/car-traffic_analysis.brf cached=false
Error (linksProcessed=0 open paths: 0): datafile E15_N45.rd5 not found
datafile E15_N45.rd5 not found
BRouter 1.6.1 / 01032020
parsed profile /home/user/workspace/brouter_original/misc/profiles2/car-traffic_analysis.brf cached=false
Error (linksProcessed=0 open paths: 0): datafile E15_N50.rd5 not found
datafile E15_N50.rd5 not found
*** WayLinker: Format a region of an OSM map for routing
cp: cannot stat '/var/www/brouter/segments4/.htaccess': No such file or directory

----- kind of off-topic -------

Btw, the misc/readmes/mapcreation.md says:

use the "ArcInfo ASCII" format

While the only choices on the SRTM download manager are:

Geo TIFF, Esri ASCII

So I guess this should be changed to Esri ASCII.

@abrensch
Copy link
Owner

uff, couple of issues....

for the compile error - my fault - just removed BPbfBlobDecoder2 from the repo - was left from an experimental POC

Which pre-processor script are you using?

The "process_pbf_planet_production.sh" is the original production script. As in your log

The old one ("process_pbf_planet.sh") will most likely not work anymore after my changes to speedup the prepropcessor.

So i propose to use the production script. But first remove the "osmupdate" part on the top, and the "scan_world" (=brouter-suspect-scan) and "traffic_simulation" calls.

And if you need help tp get the SRTM data, contact me. Or start without elevantion data for a start.

regards, Arndt

@hb0
Copy link

hb0 commented Mar 18, 2020

Hi Arndt,

looks good - a rd5 file was generated. I'll try to use it in BRouter later this day and let you know the outcome.

Thanks a lot for the fast replies.
Best Regards
Armin


Below the altered script I used to generate the rd5 file (maybe this saves others some time):

#!/bin/bash
set -e

# Added
JAVA='/usr/bin/java -Xmx6144m -Xms6144m -Xmn256m'
BROUTER_PROFILES=$(realpath "../../profiles2")
BROUTER_JAR=$(realpath $(ls ../../../brouter-server/target/brouter-server-*-jar-with-dependencies.jar))
OSMOSIS_JAR=$(realpath "../../pbfparser/osmosis.jar")
PROTOBUF_JAR=$(realpath "../../pbfparser/protobuf.jar")
PBFPARSER_JAR=$(realpath "../../pbfparser/pbfparser.jar")
PLANET_FILE=${PLANET_FILE:-$(realpath "./planet-latest.osm.pbf")} # (!) expects PLANET_FILE to be set OR 'planet-latest.osm.pbf'
SRTM_PATH=/home/user/workspace/brouter_original/misc/scripts/mapcreation/srtm

rm -rf planet-old.osm.pbf
rm -rf planet-new.osm.pbf
touch mapsnapshpttime.txt

rm -rf tmp

mkdir tmp
cd tmp
mkdir nodetiles
mkdir waytiles
mkdir waytiles55
mkdir nodes55

$JAVA -cp ${OSMOSIS_JAR}:${PROTOBUF_JAR}:${PBFPARSER_JAR}:${BROUTER_JAR} \
	-Ddeletetmpfiles=true -DuseDenseMaps=true \
	btools.util.StackSampler btools.mapcreator.OsmFastCutter \
	${BROUTER_PROFILES}/lookups.dat nodetiles waytiles nodes55 waytiles55 \
	bordernids.dat  relations.dat  restrictions.dat \
	${BROUTER_PROFILES}/all.brf ${BROUTER_PROFILES}/trekking.brf ${BROUTER_PROFILES}/softaccess.brf \
	${PLANET_FILE}

printf "\n\n---------------------- unotes55 -----------------\n\n\n"
mkdir unodes55
$JAVA -cp ${BROUTER_JAR} -Ddeletetmpfiles=true -DuseDenseMaps=true btools.util.StackSampler \
	btools.mapcreator.PosUnifier nodes55 unodes55 bordernids.dat bordernodes.dat ${SRTM_PATH}

printf "\n\n---------------------- segments -----------------\n\n\n"
mkdir segments
$JAVA -cp ${BROUTER_JAR} -DuseDenseMaps=true -DskipEncodingCheck=true btools.util.StackSampler \
	btools.mapcreator.WayLinker unodes55 waytiles55 bordernodes.dat restrictions.dat ${BROUTER_PROFILES}/lookups.dat \
	${BROUTER_PROFILES}/all.brf segments rd5

cd ..

rm -rf segments
mv tmp/segments segments
touch -r mapsnapshpttime.txt segments/*.rd5

@hb0
Copy link

hb0 commented Mar 19, 2020

I can confirm that the self-generated rd5 file works in BRouter and it contains height data.

However: The height data for the same OSM nodes is different, see screenshot:
(left: rd5 from BRouter website, right self-generates rd5 file with CIGAR SRTM 30M)

shortest_diff-heights

Thus, I'm wondering if you do pre-process the SRTM data before using it for the rd5 file generation?
I'm referring to the content of your talks at the FOSSGIS:

a) FOSSGIS 2014: Hysterese Filter on the 90M data
I thought this is only done inside the car-eco profile?
Or is this done on the CGIAR SRTM data? Or during routing, profile-independent?
This info would be good to have for documentation in the research project I'm involved in.

b) FOSSGIS 2015, minute 19:40+: Noise-Filter new
In which step is this done? inside BRouter or before generating the rd5 file?

We might use high-res height data in our research project as we only test this in a small region where this data is available but I would need to know if the data we insert into the rd5 generation is altered before it's output again in the BRouter geometry output.

@abrensch
Copy link
Owner

However: The height data for the same OSM nodes is different, see screenshot:
(left: rd5 from BRouter website, right self-generates rd5 file with CIGAR SRTM 30M)

Hi Armin,

I wasnt't aware there is a 30m update at CGIAR. And I just tried to download a file, but I got the 90m version. And I compared that with the one I downloaded years ago, and it differs:

new download:

19.09.2008 15:05 2.479 readme.txt
19.09.2008 18:31 140.400.178 srtm_38_02.asc
19.09.2008 18:31 656 srtm_38_02.prj

my version:

-rw-rw-r-- 1 abrensch abrensch 2479 Sep 19 2008 readme.txt
-rw-rw-r-- 1 abrensch abrensch 140452772 Nov 24 2008 srtm_38_02.asc
-rw-rw-r-- 1 abrensch abrensch 656 Nov 24 2008 srtm_38_02.prj

so they differ in size and timestamp. But readme says in both cases it's version 4.1

Thus, I'm wondering if you do pre-process the SRTM data before using it for the rd5 file generation?

I'm still using 90m data. Never took the 30m data productive. Just could'nt prove that it actually gives better results.

I do use a pre-processor, but this is just technical to have a compact binary format for speed parsing. I call it ".bef (brouter elevation format). But it does not apply any filtering.

Actuall I developed the bef-format in order to be able to process 30m data.

Hysterese Filter on the 90M data
I thought this is only done inside the car-eco profile?

Yes, hysteresis filter is done during routing.

In which step is this done? inside BRouter or before generating the rd5 file?

For 90m, I use a simple linear interpolation between the 4 surrounding grid-points.

For 30m, I developed the advanced filtering (but did not take it productive)

We might use high-res height data in our research project as we only test this in a small region where this data is available but I would need to know if the data we insert into the rd5 generation is altered before it's output again in the BRouter geometry output.

I also converted some lidar-data into the bef-format for >60 degree where srtm is not available.

I could give a access to my cgiar-files or my .bef-files. Just send me a mail, so I can provide an access link to some files. I cannot do that publicly due to licence issues.

regards, Arndt

@hb0
Copy link

hb0 commented Apr 7, 2020

Hi Arndt,

I'm trying to convert this DGM2 data (e.g. section 4005640) to the ArcInfo/ESRI ASCII format in order to use it with the mapcreation script.

The data comes in 2 km x 2 km XYZ text files.
The reference coordinate system EPSG: 25833 is used.

I tried the following steps:

  1. Convert the xyz-files into the WGS84 reference system as this is used by the CIGAR elevation data the script usually imports: gdalwarp -s_srs EPSG:25833 -t_srs EPSG:4326 -dstnodata -999999999.0 -r near -of GTiff dgm2_4005640/334185658_dgm2.xyz 334185658_dgm2_warp.tif

  2. Merge the converted xyz-files from (1) together into a larger batch: gdalbuildvrt -resolution highest -r nearest -input_file_list buildvrtInputFiles.txt Merged.vrt

  • I've used -dstnodata -999999999.0 in (1) in order to resolve nodata-areas with -resolution highest in (2) or else some "nodata" areas did overwrite the areas with data during merging - at least when visualized in QGIS.
  1. Convert the merged data from (2) into the ASCII format: gdal_translate -of AAIGrid Merged.vrt Converted.asc

  2. As there are very small gaps between the merged xyz file areas I'm filling this data with: gdal_fillnodata.py -md 10 -b 1 -of GTiff Converted.asc Filled.tif

  3. In order to get the ASCII format again I'm using (3) again - there is probably a better way but it seems to work out well when looking at the visualized data in QGIS after: gdal_translate -of AAIGrid Filled.tif Filled_Converted.asc

  4. Now I simply tried to renamed the output file of (5) (Filled_Converted.asc) to srtm_39_02, copied it into a zip file called misc/scripts/mapcreation/srtm/srtm_39_02.zip and pointed the mapcreation script to that folder. I'm unsure if the file name actually has a semantic meaning for BRouter? As OSM file I used an OSM file from Dresden area.

  5. But when I use the newly generated .rd5 file in BRouter in the middle of Dresden where the generated height-data should have data available the BRouter outputs non-expected heights:



{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "creator": "BRouter-1.1",
        "name": "brouter_trekking_2",
        "track-length": "271",
        "filtered ascend": "0",
        "plain-ascend": "0",
        "total-time": "44",
        "total-energy": "3481",
        "cost": "1074",
        "messages": [
          ["Longitude", "Latitude", "Elevation", "Distance", "CostPerKm", "ElevCost", "TurnCost", "NodeCost", "InitialCost", "WayTags", "NodeTags"],
          ["13741849", "51043137", "-8192", "2", "5050", "0", "0", "0", "0", "highway=footway surface=fine_gravel", ""],
          ["13741802", "51043208", "-8192", "9", "5050", "0", "0", "0", "0", "reversedirection=yes highway=footway", ""],
          ["13741694", "51043177", "-8192", "9", "1350", "0", "93", "0", "0", "reversedirection=yes highway=unclassified", ""],
          ["13741543", "51045114", "-8192", "232", "1650", "0", "95", "0", "0", "reversedirection=yes highway=secondary surface=asphalt", ""],
          ["13741648", "51045226", "-8192", "15", "1650", "0", "0", "0", "0", "highway=secondary surface=asphalt cycleway=lane oneway=yes", ""],
          ["13741690", "51045189", "-8192", "4", "5050", "0", "126", "0", "0", "reversedirection=yes highway=path", ""]
        ]
      },
      "geometry": {
        "type": "LineString",
        "coordinates": [
          [13.741857, 51.043121],
          [13.741849, 51.043137],
          [13.741802, 51.043208],
          [13.741694, 51.043177],
          [13.741611, 51.043308],
          [13.741550, 51.043400],
          [13.741371, 51.043666],
          [13.741259, 51.043885],
          [13.741221, 51.043990],
          [13.741181, 51.044142],
          [13.741170, 51.044306],
          [13.741182, 51.044465],
          [13.741201, 51.044563],
          [13.741249, 51.044666],
          [13.741302, 51.044755],
          [13.741543, 51.045114],
          [13.741648, 51.045226],
          [13.741678, 51.045199]
        ]
      }
    }
  ]
}

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "creator": "BRouter-1.1",
        "name": "brouter_shortest_2",
        "track-length": "286",
        "filtered ascend": "14011",
        "plain-ascend": "6768",
        "total-time": "2147483647",
        "total-energy": "5773953",
        "cost": "286",
        "messages": [
          ["Longitude", "Latitude", "Elevation", "Distance", "CostPerKm", "ElevCost", "TurnCost", "NodeCost", "InitialCost", "WayTags", "NodeTags"],
          ["13742241", "51043245", "-4322", "34", "1000", "0", "0", "0", "0", "highway=footway surface=fine_gravel footway=sidewalk", ""],
          ["13742152", "51043469", "-4322", "27", "1000", "0", "0", "0", "0", "reversedirection=yes highway=footway lit=unknown", ""],
          ["13741885", "51045106", "2445", "206", "1000", "0", "0", "0", "0", "highway=footway lit=unknown", ""],
          ["13741690", "51045189", "2445", "17", "1000", "0", "0", "0", "0", "reversedirection=yes highway=footway surface=asphalt", ""],
          ["13741648", "51045226", "2445", "2", "1000", "0", "0", "0", "0", "highway=path", ""]
        ]
      },
      "geometry": {
        "type": "LineString",
        "coordinates": [
          [13.741857, 51.043121],
          [13.741849, 51.043137],
          [13.741899, 51.043182, -4322.75],
          [13.742241, 51.043245, -4322.75],
          [13.742229, 51.043287, -4322.75],
          [13.742203, 51.043335, -4322.75],
          [13.742152, 51.043469, -4322.75],
          [13.742257, 51.043513, -4322.75],
          [13.742353, 51.043576, -4322.75],
          [13.742412, 51.043630, -7948.0],
          [13.742430, 51.043783, -7948.0],
          [13.742407, 51.043904, -7948.0],
          [13.742336, 51.044044, -7948.0],
          [13.742222, 51.044221, -7948.0],
          [13.742171, 51.044409, -7948.0],
          [13.742054, 51.044544, -7948.0],
          [13.742048, 51.044773, 6073.75],
          [13.741939, 51.044949, 6073.75],
          [13.741880, 51.045003, 6073.75],
          [13.741885, 51.045106, 2445.0],
          [13.741690, 51.045189, 2445.0],
          [13.741678, 51.045199, 2445.0]

A) Now I'm wondering if I simply have to make sure the height-data area needs to match the shape of the CIGAR data (5*5 degrees)? I'm currently only using an area of ~ 10 km x 8 km of height data of inner Dresden region.

B) Or if I missed an obvious step somewhere in the process.

In the following picture you can see the CIGAR SRTM data (the outer, red data) and the converted DGM2 data (inner, black area) which I generated and tried to use:

Screenshot from 2020-04-07 09-46-06

@abrensch
Copy link
Owner

abrensch commented Apr 7, 2020

A) Now I'm wondering if I simply have to make sure the height-data area needs to match the shape of
the CIGAR data (5*5 degrees)? I'm currently only using an area of ~ 10 km x 8 km of height data of
inner Dresden region.

not sure, really.

Actually, I never used the asc/zip Parser for anything else then the 90 meter, 5 times 5 degrees, 6001
times 6001 Pixel files.

Though looking at the parser:

Method "readFromStream" of https://github.com/abrensch/brouter/blob/master/brouter-map-creator/src/main/java/btools/mapcreator/SrtmData.java

and the linear-interpolation formular:

Method " getElevation" of https://github.com/abrensch/brouter/blob/master/brouter-map-creator/src/main/java/btools/mapcreator/SrtmRaster.java

It should work if the header (rows/cols, lower left corner, cellsize) in your ASC really match the structure o your data.

But my guess it that it does not match, the "interpolation" formular will happily extrapolate and you get nonsense results.

@hb0
Copy link

hb0 commented Apr 9, 2020

Thanks a lot for pointing to the relevant Code sections, this helped.

Using gdalinfo generatedFile.asc I noticed that my generated height Z-data is of type Float but in the code the data is read as Short type in the SrtmData.readFromStream():

short val = n < -250 ? Short.MIN_VALUE : (short) (n);
user@HBWS:~/Downloads/_BIK$ gdalinfo Filled_Converted.asc 
Driver: AAIGrid/Arc/Info ASCII Grid
Files: Filled_Converted.asc
Size is 9754, 3907
Coordinate System is `'
Origin = (13.629384265679001,51.085910071442996)
Pixel Size = (0.000023640350000,-0.000023640350000)
Corner Coordinates:
Upper Left  (  13.6293843,  51.0859101) 
Lower Left  (  13.6293843,  50.9935472) 
Upper Right (  13.8599722,  51.0859101) 
Lower Right (  13.8599722,  50.9935472) 
Center      (  13.7446783,  51.0397286) 
Band 1 Block=9754x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-1000000000

Looking into the generated .asc file itself confirms this:

user@HBWS:~/Downloads/_BIK$ head Filled_Converted.asc 
ncols        9754
nrows        3907
xllcorner    13.629384265679
yllcorner    50.993547223993
cellsize     0.000023640350
NODATA_value  -1000000000
 -1000000000.0 -1000000000 [....] -1000000000 244.9199981689453125 244.92999267578125 244.970001220703125 244.970001220703125 244.9799957275390625 245.1199951171875 245.1199951171875 245.07000732421875 245 244.9799957275390625 [...] -1000000000 [...]
[...]

Thus, I'll use something like -999 as noDataValue to be within the short range.

And I'll either convert the float z-value (height) into short as we don't have heights above 3000 meters here in Germany or I'll check if I can change the BRouter code to work with float to not loose the precision. I'll report back for others who might try this in future.

@hb0
Copy link

hb0 commented Apr 9, 2020

Short update:
I tried to converting the height z-values to Int16 / Short which now produces more meaningful data:

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "creator": "BRouter-1.1",
        "name": "brouter_shortest_0",
        "track-length": "251",
        "filtered ascend": "0",
        "plain-ascend": "0",
        "total-time": "173",
        "total-energy": "632",
        "cost": "251",
        "messages": [
          ["Longitude", "Latitude", "Elevation", "Distance", "CostPerKm", "ElevCost", "TurnCost", "NodeCost", "InitialCost", "WayTags", "NodeTags"],
          ["13741849", "51043137", "113", "2", "1000", "0", "0", "0", "0", "highway=footway surface=fine_gravel footway=sidewalk", ""],
          ["13741768", "51043262", "113", "16", "1000", "0", "0", "0", "0", "reversedirection=yes highway=footway footway=crossing crossing=uncontrolled", ""],
          ["13741690", "51045189", "113", "231", "1000", "0", "0", "0", "0", "reversedirection=yes highway=footway footway=sidewalk", ""],
          ["13741648", "51045226", "113", "2", "1000", "0", "0", "0", "0", "highway=path", ""]
        ]
      },
      "geometry": {
        "type": "LineString",
        "coordinates": [
          [13.741857, 51.043121, 113.0],
          [13.741849, 51.043137, 113.0],
          [13.741802, 51.043208, 113.0],
          [13.741768, 51.043262, 113.0],
          [13.741730, 51.043273, 113.0],
          [13.741640, 51.043427, 113.0],
          [13.741420, 51.043805, 113.0],
          [13.741327, 51.044006, 113.0],
          [13.741291, 51.044140, 113.0],
          [13.741269, 51.044308, 113.0],
          [13.741284, 51.044462, 112.75],
          [13.741303, 51.044551, 113.0],
          [13.741336, 51.044643, 113.0],
          [13.741634, 51.045099, 113.0],
          [13.741690, 51.045189, 113.0],
          [13.741678, 51.045199, 113.25]
        ]
      }
    }
  ]
}

Here is the script I used for conversion - maybe it safes others some time:

#!/bin/bash -e
#
# Converts '.xyz' data downloaded from https://www.geodaten.sachsen.de/downloadbereich-dgm2-4164.html
# into 'AAIGrid/Arc/Info ASCII Grid' readable by the BRouter/btools/mapcreation script to generate .rd5 data.
#
# author Armin
# version 1.0.0
# since 1.0.0

#
# Static constants which are required by multiple methods
#
# Input/output paths
DGM_FOLDER='dgm2_4005640'
OUTPUT_FOLDER='output'
OUTPUT_FILE_NAME='srtm_39_02' # btools/mapcreation expects it to be in the srtm nameing format
#
# Temporary files
TEMP_FOLDER='tmp' # Folder to be used for temporary files
XYZ_FILES_LIST="$TEMP_FOLDER/xyz-files.txt" # File to write the files names of the .xyz files to
TIF_FILES_LIST="$TEMP_FOLDER/tif-files.txt" # File to write the files names of the .tif files to
MERGED_DATA_FILE="$TEMP_FOLDER/merged.vrt" # File to write the merged data to
CONVERTED_DATA_FILE="$TEMP_FOLDER/merged.asc" # File to write the converted data to
FILLED_GAPS_FILE="$TEMP_FOLDER/filled.tif" # File to write the data with filled gaps to
FILLED_CONVERTED_FILE="$TEMP_FOLDER/filled.asc" # .asc-File to write the data with filled gaps to

#
# The starting point of this script
#
# $@: Injected command-line arguments. Can be accessed via $1, $2, etc. if existing.
main() {

  prepare $TEMP_FOLDER $OUTPUT_FOLDER
  readXyzFiles $DGM_FOLDER $XYZ_FILES_LIST
  convertCoordinateSystem $DGM_FOLDER $XYZ_FILES_LIST $TIF_FILES_LIST
  mergeFiles $TIF_FILES_LIST $MERGED_DATA_FILE

  # We convert, fillGaps and convert again.
  # Skipping the first convert generated data which contained erroneous data (when visualized in QGIS)
  convert $MERGED_DATA_FILE $CONVERTED_DATA_FILE
  fillGaps $CONVERTED_DATA_FILE $FILLED_GAPS_FILE
  convert $FILLED_GAPS_FILE $FILLED_CONVERTED_FILE

  assempleOutput $FILLED_CONVERTED_FILE $OUTPUT_FOLDER $OUTPUT_FILE_NAME

  printf "\nDone.\n"
}

# Zips the converted data and adds meta files
#
# $1: File containing the data to be used
# $2: Name part of the file to write the data to
# $3: Folder to write the output data to
assempleOutput() {
  INPUT_FILE=$1
  OUTPUT_FOLDER=$2
  OUTPUT_FILE_NAME=$3
  README_FILE="readme.txt"
  PRJ_TEMPLATE_FILE="template.prj"

  printf "\nAssembling $INPUT_FILE -> $OUTPUT_FOLDER/$OUTPUT_FILE_NAME.zip ...\n"
  cd $TEMP_FOLDER || exitWithError "assempleOutput(): Temp folder not found: $TEMP_FOLDER"
  echo "This archive contains converted data from https://www.geodaten.sachsen.de/downloadbereich-dgm2-4164.html" > $README_FILE
  cp ../$PRJ_TEMPLATE_FILE $OUTPUT_FILE_NAME.prj || exitWithError "assempleOutput(): Copying .prj file failed"
  zip \
    $OUTPUT_FILE_NAME.zip $(basename $INPUT_FILE) $OUTPUT_FILE_NAME.prj $README_FILE \
    || exitWithError "assempleOutput(): Zipping failed"
  mv $OUTPUT_FILE_NAME.zip ../$OUTPUT_FOLDER || exitWithError "assempleOutput(): Moveing zip file failed"
  cd ..
}

# Fill gaps in merged data
#
# $1: File containing the data to fill the gaps of
# $2: File to write the enhanced data to
fillGaps() {
  INPUT_FILE=$1
  OUTPUT_FILE=$2
  OUTPUT_TYPE="GTiff" # format to save the data to. Using AAIGrid did produce empty data.
  MAX_PIXEL_DISTANCE="10" # maximum gap-distance in px that will be filled
  BAND_TO_FILL="1" # the "band" to be filled. Usually our data only contains one "band".

  printf "\nFilling gaps in $INPUT_FILE -> $OUTPUT_FILE ...\n"
  gdal_fillnodata.py \
    -md $MAX_PIXEL_DISTANCE -b $BAND_TO_FILL \
    -of $OUTPUT_TYPE $INPUT_FILE $OUTPUT_FILE || exitWithError "fillGaps(): gdal_fillnodata.py failed"
}

# Convert the input file to AAIGrid (.asc)
#
# $1: File containing the merged data
# $2: File to write the converted data to
convert() {
  INPUT_FILE=$1
  OUTPUT_FILE=$2
  OUTPUT_DATA_TYPE="Int16" # Convert the float input data type as btools/mapcreation reads as `Short`
  CONVERT_TO="AAIGrid" # format to convert the data to - AAIGrid can be read by btools/mapcreation

  printf "\nConverting $INPUT_FILE -> $OUTPUT_FILE ...\n"
  gdal_translate \
    -ot $OUTPUT_DATA_TYPE -of $CONVERT_TO $INPUT_FILE $OUTPUT_FILE \
    || exitWithError "convert(): gdal_translate command failed"
}

# Merge the input files
#
# $1: File which contains the list of tif files
# $2: File to write the merged data to
mergeFiles() {
  INPUT_FILES_LIST=$1
  OUTPUT_FILE=$2
  RESOLUTION="highest" # picks the smallest pixel-raster if it differes between the input files
  RESAMPLING_METHOD="nearest" # default, fastest algorithm, worst interpolation quality

  printf "\nMerging $INPUT_FILES_LIST -> $OUTPUT_FILE ...\n"
  gdalbuildvrt \
    -resolution $RESOLUTION -r $RESAMPLING_METHOD \
    -input_file_list $INPUT_FILES_LIST $OUTPUT_FILE || exitWithError "mergeFiles(): gdalbuildvrt command failed"
}

# Convert XYZ files to EPSG:4326 (WGS84) - outputs .tif files
#
# $1: Name of folder containing xyz files
# $2: File which contains the list of xyz files - which contain the input data
# $3: File which contains the list of tif files - which contain the converted data
convertCoordinateSystem() {
  INPUT_FOLDER=$1
  INPUT_FILES_LIST=$2
  OUTPUT_FILES_LIST=$3
  FROM_EPSG="25833"
  TO_EPSG="4326"
  NO_DATA_VALUE="-999"
  RESAMPLING_METHOD="near" # default, fastest algorithm, worst interpolation quality

  # This line reading can have side-effects, see: https://stackoverflow.com/a/1521498/5815054
  printf "\nconvertCoordinateSystem of $INPUT_FILES_LIST ...\n"
  while read FILE; do
    OUTPUT_FILE=$(echo $FILE | sed --expression 's/.xyz/.tif/g' | sed -e "s/$INPUT_FOLDER/$TEMP_FOLDER/g")
    echo $OUTPUT_FILE >> $OUTPUT_FILES_LIST

    printf "$FILE -> $OUTPUT_FILE\n"
    gdalwarp -q \
      -s_srs EPSG:$FROM_EPSG -t_srs EPSG:$TO_EPSG \
      -dstnodata $NO_DATA_VALUE -r $RESAMPLING_METHOD \
      -of GTiff $FILE $OUTPUT_FILE || exitWithError "convertCoordinateSystem(): gdalwarp command failed"
  done <$INPUT_FILES_LIST
}

# Searches for xyz files
#
# $1: Name of folder containing xyz files
# $2: File to write the list of files found to (format: one file path per line)
readXyzFiles() {
  INPUT_FOLDER=$1
  OUTPUT_FILES_LIST=$2
  FIND_COMMAND='find dgm2_4005640/ -type f -name "*.xyz"'

  eval $FIND_COMMAND > $OUTPUT_FILES_LIST || exitWithError "readXyzFiles(): find command failed"

  COUNT=$(eval $FIND_COMMAND | wc -l)
  printf "\nxyz-files found in ./$INPUT_FOLDER/: $COUNT\n"
}

# Ensures this execution is independent from last execution temporary files and creates folders
#
# $1: Name of the folder to clean up or create for temporary file
# $2: Name of the folder to create for the output data
prepare() {
  CLEANUP_FOLDER=$1
  OUTPUT_FOLDER=$2

  printf "\nPreparing ./$CLEANUP_FOLDER/ and ./$OUTPUT_FOLDER/ ...\n"
  rm -rf $CLEANUP_FOLDER || exitWithError "cleanup(): rm command failed"
  mkdir $CLEANUP_FOLDER || exitWithError "cleanup(): mkdir command failed"
  mkdir -p $OUTPUT_FOLDER || exitWithError "cleanup(): mkdir command failed"
}

# Interrupts the script execution
#
# $1: The error message to be shown before exiting
exitWithError() {
  SCRIPT_NAME=$(basename $0)

  printf "\n${SCRIPT_NAME}: ${1:-"Unknown Error"}\n\n" 1>&2
  exit 1
}

# Executes the main() function as starting point of this script
main "$@"

I will probably also try to adjust the btools/mapcreation code to work with the floating point precision of the original DGM2-data but for now that's it.

@abrensch
Copy link
Owner

abrensch commented Apr 9, 2020

I will probably also try to adjust the btools/mapcreation code
to work with the floating point precision of the original
DGM2-data but for now that's it.

Sure you can adapt the parse to handle floating point.

Then you proably also wnt to change the data-type of "eval_array", or at least move the "*4" multiplication from the interpolation code to the parser.

Finally, for what's going into the rd5-files this is a "short" datatype with 25cm resolution. Pretty hard to change that. But 25cm more or less matches the accuracy of the data (15cm according to Landesvermessungsamt Sachsen), so it should be o.k....

@hb0
Copy link

hb0 commented Aug 13, 2020

I could give a access to my cgiar-files or my .bef-files. Just send me a mail, so I can provide an access link to some files. I cannot do that publicly due to licence issues.

@abrensch Could you please send me the download link for your cgiar files?
(Update: Sorry, I just sent you the email you requested, I did not see this request before.)

I can then generate a rd5 file with the script posted above which imports the srtm/srtm_**.zip.


For others who want to use un-smoothened DGM2 data, this is why:
I've troubles with the DGM2 data as it's not smoothed out, e.g. when a bike lane goes through an underpass of an elevated railway trail the cycleway contains a "4 meter step" because ofc the correct height of the way in the tunnel cannot be calculated.

Screenshot from 2020-08-13 14-06-33

In the online version of the BRouter this problem does not occur, so I'll just use the height data from there for now:
Screenshot from 2020-08-13 14-10-57

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants