Skip to content
This repository has been archived by the owner on Dec 20, 2018. It is now read-only.

ERROR org.apache.spark.executor.Executor ... scala.MatchError: Some() at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo #2

Closed
aklink opened this issue Oct 4, 2016 · 14 comments
Closed

ERROR org.apache.spark.executor.Executor ... scala.MatchError: Some() at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo #2

aklink opened this issue Oct 4, 2016 · 14 comments
Assignees
@aklink
Labels
ready

Comments

@aklink
Copy link
Member

aklink commented Oct 4, 2016 •
edited by vsimko

When I try:

java -cp target/biggis-landuse-0.0.1-SNAPSHOT.jar
biggis.landuse.spark.examples.GeotiffToPyramid
./data/DOP_RGBI_T2.tif
new_layer
./data/pyramid

I get the following error message:

15:58:50 ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 0.0 (TID 0)
scala.MatchError: Some() (of class scala.Some)
at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo(GeoTiffCSParser.scala:570)
at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.createGeoTiffCSParameters(GeoTiffCSParser.scala:172)
at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.geoTiffCSParameters$lzycompute(GeoTiffCSParser.scala:78)
at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.geoTiffCSParameters(GeoTiffCSParser.scala:78)
at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.model(GeoTiffCSParser.scala:80)
at geotrellis.raster.io.geotiff.tags.TiffTags.crs$lzycompute(TiffTags.scala:207)
at geotrellis.raster.io.geotiff.tags.TiffTags.crs(TiffTags.scala:205)
at geotrellis.raster.io.geotiff.reader.GeoTiffReader$.readGeoTiffInfo(GeoTiffReader.scala:315)
at geotrellis.raster.io.geotiff.reader.GeoTiffReader$.readSingleband(GeoTiffReader.scala:67)
at geotrellis.raster.io.geotiff.reader.GeoTiffReader$.readSingleband(GeoTiffReader.scala:61)
at geotrellis.raster.io.geotiff.SinglebandGeoTiff$.apply(SinglebandGeoTiff.scala:40)
at geotrellis.spark.io.hadoop.formats.GeotiffInputFormat.read(GeotiffInputFormat.scala:28)
at geotrellis.spark.io.hadoop.formats.BinaryFileInputFormat$$anonfun$createRecordReader$1.apply(BinaryFileInputFormat.scala:34)
at geotrellis.spark.io.hadoop.formats.BinaryFileInputFormat$$anonfun$createRecordReader$1.apply(BinaryFileInputFormat.scala:34)
at geotrellis.spark.io.hadoop.formats.BinaryFileRecordReader.initialize(BinaryFileInputFormat.scala:18)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.(NewHadoopRDD.scala:158)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:129)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:64)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
@aklink
Copy link
Member Author

aklink commented Oct 4, 2016

Issue is specific to used image.
Trying a landsat image (created with ERDAS Imagine) is ok (no issue).

Trying a different aerila image leads to different error:

15:05:03 ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 3.0 (TID 3)
geotrellis.raster.GeoAttrsError: invalid cols: 0
at geotrellis.raster.RasterExtent.(RasterExtent.scala:72)
at geotrellis.raster.GridDefinition$class.toRasterExtent(GridDefinition.scala:36)
at geotrellis.raster.GridExtent.toRasterExtent(GridExtent.scala:9)
at geotrellis.raster.reproject.ReprojectRasterExtent$.apply(ReprojectRasterExtent.scala:105)
at geotrellis.spark.reproject.TileRDDReproject$$anonfun$6$$anonfun$apply$2.apply(TileRDDReproject.scala:185)
at geotrellis.spark.reproject.TileRDDReproject$$anonfun$6$$anonfun$apply$2.apply(TileRDDReproject.scala:182)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

@aklink
Copy link
Member Author

aklink commented Oct 4, 2016

The first issue with "geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo" is caused by an older SRS (Gauss-Krüger Zone 3, used in Baden-Württemberg) in GeoTiff file, can be fixed by applying "gdalwarp -t_srs EPSG:25832 srcfile targetfile" to file to use a newer SRS (ETRS89 UTM32N). Geotrellis seems not to support older SRS.

The second issue with "geotrellis.raster.RasterExtent.(RasterExtent.scala:72)" is caused by a missing SRS in GeoTiff file, can be fixed by applying "gdal_translate -a_srs EPSG:25832 srcfile targetfile" in advance to add a valid SRS (on test file, not in sourcecode).

@aklink aklink closed this as completed Oct 4, 2016
@aklink
Copy link
Member Author

aklink commented Oct 13, 2016

Maybe related to this issues?

@aklink
Copy link
Member Author

aklink commented Oct 13, 2016

Problematic CRS is EPSG:31467 in following GDAL representation:

PROJCS["DHDN_3_Degree_Gauss_Zone_3",
    GEOGCS["GCS_Deutsches_Hauptdreiecksnetz",
        DATUM["Deutsches_Hauptdreiecksnetz",
            SPHEROID["Bessel_1841",6377397.155,299.1528128,
                AUTHORITY["EPSG","7004"]],
            AUTHORITY["EPSG","6314"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",9],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",3500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]]]

@aklink aklink reopened this Oct 13, 2016
@vsimko
Copy link
Member

vsimko commented Oct 14, 2016

It works for my TIF file in EPSG_31467:

$ gdalinfo morning_EPSG_31467.tif
Driver: GTiff/GeoTIFF
Files: morning_EPSG_31467.tif
Size is 5161, 7911
Coordinate System is:
PROJCS["DHDN / 3-degree Gauss-Kruger zone 3",
    GEOGCS["DHDN",
        DATUM["Deutsches_Hauptdreiecksnetz",
            SPHEROID["Bessel 1841",6377397.155,299.1528128000008,
                AUTHORITY["EPSG","7004"]],
            TOWGS84[598.1,73.7,418.2,0.202,0.045,-2.455,6.7],
            AUTHORITY["EPSG","6314"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4314"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",9],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",3500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AUTHORITY["EPSG","31467"]]
Origin = (3445385.870000000111759,5447456.839999999850988)
Pixel Size = (5.000000000000000,-5.000000000000000)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_IMAGEDESCRIPTION=
  TIFFTAG_RESOLUTIONUNIT=3 (pixels/cm)
  TIFFTAG_XRESOLUTION=100
  TIFFTAG_YRESOLUTION=100
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( 3445385.870, 5447456.840) (  8d15' 3.88"E, 49d 9'46.43"N)
Lower Left  ( 3445385.870, 5407901.840) (  8d15'23.00"E, 48d48'25.95"N)
Upper Right ( 3471190.870, 5447456.840) (  8d36'17.74"E, 49d 9'52.74"N)
Lower Right ( 3471190.870, 5407901.840) (  8d36'27.83"E, 48d48'32.18"N)
Center      ( 3458288.370, 5427679.340) (  8d25'48.14"E, 48d59' 9.82"N)
Band 1 Block=5161x1 Type=Byte, ColorInterp=Palette
  Color Table (RGB with 256 entries)
...

@aklink
Copy link
Member Author

aklink commented Oct 14, 2016

Seems like the major difference between the two CRS representations is missing
AUTHORITY["EPSG","31467"] as well as TOWGS84[598.1,73.7,418.2,0.202,0.045,-2.455,6.7]
in non-working variant (unfortunatelly all tiles in BW are using this non-working representation)

@pomadchin
Copy link

pomadchin commented Nov 6, 2016 •
edited

Is this still an issue? Can you throw us a non working example, or (as i can understand) the problem is in corrupted (or specific) geotiff metadata? Or it is locationtech/geotrellis#1415 ?

@aklink
Copy link
Member Author

aklink commented Nov 7, 2016

I am not sure if this exactly issue locationtech/geotrellis#1415 ...

However it is specific to SRS MetaData of DHDN_3_Degree_Gauss_Zone_3.

When looking on: http://spatialreference.org/ref/epsg/31467/
It is neither exactly http://spatialreference.org/ref/epsg/31467/ogcwkt/
nor http://spatialreference.org/ref/epsg/31467/esriwkt/ due to a slighty different string representation.

Please find attached file using DHDN_3_Degree_Gauss_Zone_3.
DOP_RGBI_T2_400x400_epsg31467.zip

@pomadchin
Copy link

pomadchin commented Nov 7, 2016 •
edited

@aklink thx for attached file, so yes, i believe it is unsupported projection, good info, think it would be fixed (or we'll try to fix it :D) in terms of locationtech/geotrellis#1415 looks like a bug in a proj4j or in tiff metadata.

@aklink
Copy link
Member Author

aklink commented Nov 21, 2016

@pomadchin Thanks for looking into this issue. I really appreciate it and I am looking forward to have this projection supported by Geotrellis.

@pomadchin pomadchin mentioned this issue Nov 21, 2016
Closed
@vsimko vsimko added the ready label Dec 22, 2016
@pomadchin
Copy link

Yo guyz, can you try this PR locationtech/geotrellis#2403 if it's not too late? :D basically it was a minor bug, sry that it took us half a year to fix it.

@pomadchin pomadchin mentioned this issue Oct 3, 2017
Merged
@aklink
Copy link
Member Author

aklink commented Oct 4, 2017

@pomadchin: Thanks for the update. I will have a look into it.
Anyhow, I have trouble building it using ./scripts/publish-local.sh in Win10.

Downloading sbt launcher for 0.13.16:
  From  http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.16/sbt-launch.jar
    To  /home/ak/.sbt/launchers/0.13.16/sbt-launch.jar
Invalid maximum heap size: -Xmx2g
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Is there any Maven repository with an already built test version? Or another simple option how to test it ontop of an "older" Geotrellis version?

@pomadchin
Copy link

@aklink sure, we have a location tech snapshot repo:

resolvers ++= Seq(
  "LocationTech GeoTrellis Snapshots" at "https://repo.locationtech.org/content/repositories/geotrellis-snapshots"
)

After that just use 1.2.0-SNAPSHOT as a GeoTrellis version in your deps.

@aklink
Copy link
Member Author

aklink commented Nov 15, 2017

@pomadchin
Sorry for my late reply. Thanks! I have tested it with 1.2.0-RC1 and it seems to work fine.
I did not check the georeferenced position yet if it is on the proper location, but at least the error is gone.
Layer is created properly.

@aklink aklink closed this as completed Nov 18, 2017
aklink added a commit that referenced this issue Nov 18, 2017
@aklink
Experimental: MultibandLayerResample (still non-working #2)
17ae31d
@aklink aklink mentioned this issue Jun 19, 2018
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@aklink aklink

Labels
ready
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pomadchin @vsimko @aklink