Optional mask #56

vincentsarago · 2019-03-04T17:43:12Z

This PR refactor the cli and cog_translate to make the internal mask optional.

We updated the documentations to add notes about overview levels, nodata/alpha handling and internal tile size.

Breaking change

This PR adds no main breaking changes:

internal mask creation is now optional (--add-mask).
internal nodata or alpha channel can be forwarding to the output dataset

ghost · 2019-04-05T15:26:45Z

This change seems to have broken my workflow. Can you help me troubleshoot? There is a good chance the cause is a poorly constructed input file but I'm not sure where to look.

Input file is an Uint8 RGBNA geotif (red, green, blue, nir, alpha).
With an older version and the command below, an RGBA COG was produced with an expected alpha mask.
rio cogeo 8bit.RGBNA.tif out.tif -b 1,2,3 --cog-profile jpeg --alpha 5 --co JPEG_QUALITY=70
With the newer version, when I run the command below, an RGBA COG is produced but the alpha mask is all 255 (all pixels valid).
rio cogeo create 8bit.RGBNA.tif out.tif -b 1,2,3 --cog-profile jpeg --add-mask --co JPEG_QUALITY=70
With the newer version, when I run the command below, an RGB COG is produced with no alpha channel.
rio cogeo create 8bit.RGBNA.tif out.tif -b 1,2,3 --cog-profile jpeg --co JPEG_QUALITY=70
With the newer version, when I run the command below, an RGBA COG is produced with an expected alpha mask.
rio cogeo create 8bit.RGBNA.tif out.tif -b 1,2,3,5 --cog-profile raw

ubuntugis/ubuntugis-unstable
GDAL: 2.4 (I think)
Is there a command to tell me which version of rio-cogeo I'm running?

vincentsarago · 2019-04-05T15:38:20Z

Hi @jaredairbusaerial,
Can you share the result of gdalinfo on your 8bit.RGBNA.tif file ?

Is there a command to tell me which version of rio-cogeo I'm running?

I'll add a --version options

ghost · 2019-04-05T17:31:44Z

Driver: GTiff/GeoTIFF
Files: IMG_SPOT6_PMS_201903211650184_ORT_3906329101.8bit.TIF
Size is 35410, 80804
Coordinate System is:
PROJCS["WGS 84 / UTM zone 15N",
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0,
AUTHORITY["EPSG","8901"]],
UNIT["degree",0.0174532925199433,
AUTHORITY["EPSG","9122"]],
AUTHORITY["EPSG","4326"]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",-93],
PARAMETER["scale_factor",0.9996],
PARAMETER["false_easting",500000],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]],
AXIS["Easting",EAST],
AXIS["Northing",NORTH],
AUTHORITY["EPSG","32615"]]
Origin = (257272.500000000000000,4553104.500000000000000)
Pixel Size = (1.500000000000000,-1.500000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 257272.500, 4553104.500) ( 95d53'24.08"W, 41d 5'34.73"N)
Lower Left ( 257272.500, 4431898.500) ( 95d50'36.53"W, 40d 0' 8.69"N)
Upper Right ( 310387.500, 4553104.500) ( 95d15'29.33"W, 41d 6'25.59"N)
Lower Right ( 310387.500, 4431898.500) ( 95d13'18.32"W, 40d 0'57.63"N)
Center ( 283830.000, 4492501.500) ( 95d33'11.24"W, 40d33'18.30"N)
Band 1 Block=5056x3328 Type=Byte, ColorInterp=Gray
Band 2 Block=5056x3328 Type=Byte, ColorInterp=Undefined
Band 3 Block=5056x3328 Type=Byte, ColorInterp=Undefined
Band 4 Block=5056x3328 Type=Byte, ColorInterp=Undefined
Band 5 Block=5056x3328 Type=Byte, ColorInterp=Alpha

vincentsarago · 2019-04-05T17:58:26Z

🤔 I was hoping not to see ColorInterp=Alpha. are you able to share the data itself @jaredairbusaerial ?

ghost · 2019-04-05T18:28:22Z

Let me try to find a data set I can process with phase one of my workflow (which converts 16bit to 8bit and adds the alpha channel). And then I'll share that. Why were you not wanting to see ColorInterp=Alpha?

vincentsarago · 2019-04-05T18:30:56Z

basically the band 5 is tag with alpha so from my understanding https://github.com/cogeotiff/rio-cogeo/blob/master/rio_cogeo/cogeo.py#L123 should read the Alpha band and create the mask. I'm testing it locally with some other dataset but it will help to work on yours ;-)

ghost · 2019-04-05T18:54:37Z

It looks like when the source file is this, things work as expected. Let me set that flag on my original data set and see if that works.
Band 1 Block=160x224 Type=Byte, ColorInterp=Red
Mask Flags: PER_DATASET ALPHA
Band 2 Block=160x224 Type=Byte, ColorInterp=Green
Mask Flags: PER_DATASET ALPHA
Band 3 Block=160x224 Type=Byte, ColorInterp=Blue
Mask Flags: PER_DATASET ALPHA
Band 4 Block=160x224 Type=Byte, ColorInterp=Alpha

vincentsarago · 2019-04-05T18:55:57Z

ah!
I'm starting to understand.

gdalinfo rgbna.tif
Driver: GTiff/GeoTIFF
Files: rgbna.tif
Size is 19584, 19584
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (145.722656250000000,14.941406250000000)
Pixel Size = (0.000004487879136,-0.000004487879136)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  ( 145.7226562,  14.9414062) (145d43'21.56"E, 14d56'29.06"N)
Lower Left  ( 145.7226562,  14.8535156) (145d43'21.56"E, 14d51'12.66"N)
Upper Right ( 145.8105469,  14.9414062) (145d48'37.97"E, 14d56'29.06"N)
Lower Right ( 145.8105469,  14.8535156) (145d48'37.97"E, 14d51'12.66"N)
Center      ( 145.7666016,  14.8974609) (145d45'59.77"E, 14d53'50.86"N)
Band 1 Block=19584x1 Type=Byte, ColorInterp=Gray
Band 2 Block=19584x1 Type=Byte, ColorInterp=Undefined
Band 3 Block=19584x1 Type=Byte, ColorInterp=Undefined
Band 4 Block=19584x1 Type=Byte, ColorInterp=Undefined
Band 5 Block=19584x1 Type=Byte, ColorInterp=Alpha

so here we have a similar file. the 5th band is a alpha band and I've set the ColorInterp to alpha with rio edit-info --colorinterp undefined,undefined,undefined,undefined,alpha rgbna.tif

but when I read the data the alpha info is not translated to mask

with rasterio.open("rgbna.tif") as src_dst:
    print(src_dst.colorinterp)
    print(src_dst.mask_flag_enums)

(<ColorInterp.gray: 1>, <ColorInterp.undefined: 0>, <ColorInterp.undefined: 0>, <ColorInterp.undefined: 0>, <ColorInterp.alpha: 6>)
([<MaskFlags.all_valid: 1>], [<MaskFlags.all_valid: 1>], [<MaskFlags.all_valid: 1>], [<MaskFlags.all_valid: 1>], [<MaskFlags.all_valid: 1>])

vincentsarago · 2019-04-05T18:56:40Z

posting the same, at the same time 👌

vincentsarago · 2019-04-05T18:58:28Z

yep the bug is there https://github.com/mapbox/rasterio/blob/master/rasterio/_io.pyx#L707-L757

vincentsarago · 2019-04-05T19:01:40Z

@sgillies so this is a question that came back multiple time

do you think it could be wise to add something like

        # use Alpha band if available even if nodata is shadowing
        elif self.colorinterp[-1] == ColorInterp.alpha:
            return self.read_masks(self.indexes[-1], **kwargs)

in https://github.com/mapbox/rasterio/blob/master/rasterio/_io.pyx#L747-L749

vincentsarago · 2019-04-05T19:02:51Z

🤦‍♂️ we already talked about it in rasterio/rasterio#1456

ghost · 2019-04-05T20:06:00Z

I'm not sure I totally understand. Are you saying the bug is in the reading of the file in cogeo? Or in the creation of the source tif (I am using rasterio libs)? However, I tried moving the alpha channel to band 4 but I have the same result when running cogeo.

Band 1 Block=26925x1 Type=Byte, ColorInterp=Gray
Band 2 Block=26925x1 Type=Byte, ColorInterp=Undefined
Band 3 Block=26925x1 Type=Byte, ColorInterp=Undefined
Band 4 Block=26925x1 Type=Byte, ColorInterp=Alpha
Band 5 Block=26925x1 Type=Byte, ColorInterp=Undefined

vincentsarago · 2019-04-05T20:08:24Z

the bug is Rasterio dataset_mask won't consider the 5th band as alpha except if the Mask Flags: PER_DATASET is set for band 1-4.
let me check something

vincentsarago · 2019-04-05T20:35:53Z

note the proposed rasterio fix in #56 (comment) won't work because rasterio._io.read_masks won't consider the band as a mask because the mask_enums is not set.

ghost · 2019-04-05T20:46:03Z

Struggling to figure out how to set the flags here... I'm pretty new to this.

vincentsarago · 2019-04-05T20:48:08Z

@jaredairbusaerial sadly I can't find a way to update the mask flag. as a quick fix, you should create a tmp file with only 4 bands

gdal_translate rgbna_alpha.tif rgb_alpha.tif -b 1 -b 2 -b 3 -b 5

I'll think about re-adding the --alpha options but I removed it because it was really prone to misusage

vincentsarago added 2 commits March 4, 2019 13:34

make internal mask optional

23bf396

add mock

3e55567

vincentsarago merged commit f9ba949 into master Mar 4, 2019

vincentsarago deleted the optional_mask branch March 4, 2019 19:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optional mask #56

Optional mask #56

vincentsarago commented Mar 4, 2019 •

edited

ghost commented Apr 5, 2019 •

edited by ghost

vincentsarago commented Apr 5, 2019

ghost commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

ghost commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

ghost commented Apr 5, 2019 •

edited by ghost

vincentsarago commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

ghost commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

ghost commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

Optional mask #56

Optional mask #56

Conversation

vincentsarago commented Mar 4, 2019 • edited

Breaking change

ghost commented Apr 5, 2019 • edited by ghost

vincentsarago commented Apr 5, 2019

ghost commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

ghost commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

ghost commented Apr 5, 2019 • edited by ghost

vincentsarago commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

ghost commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

ghost commented Apr 5, 2019

vincentsarago commented Apr 5, 2019

vincentsarago commented Mar 4, 2019 •

edited

ghost commented Apr 5, 2019 •

edited by ghost

ghost commented Apr 5, 2019 •

edited by ghost