Uploading imagery to old MongoDB API manually

[spwoodcock]

• For Freetown we have a 12GB GeoTIFF. The AWS Batch job fails processing this - it's too large.
• Instead we can do a manual insert, until we overhaul with the new uploader.

1. Download Source

curl -L "https://drive.usercontent.google.com/download?id=FILE_ID&export=download&confirm=t" -o freetown.tif

2. Convert to COG

docker run --rm --entrypoint=sh ghcr.io/osgeo/gdal:alpine-small-latest

# Check compression
gdalinfo freetown.tif | grep Compression

# Convert to COG, keeping same source quality and compression type, e.g.
gdal_translate -of COG -co COMPRESS=JPEG -co QUALITY=75 -co BLOCKSIZE=256 \
  -co BIGTIFF=YES \
  All_Freetown_Urban_with_senstive_Areas_downrezzed_with_overviews.tif \
  freetown_cog.tif

# OR Convert to COG, keeping source compression
gdal_translate -of COG -co COMPRESS=COPY -co BLOCKSIZE=512 \
  -co COPY_SRC_OVERVIEWS=YES \
  -co BIGTIFF=YES \
  freetown.tif freetown_cog.tif

3. Upload to S3

aws s3 cp freetown_cog.tif s3://oam-uploader-production-temp/freetown_cog.tif
aws s3api put-object-acl --bucket oam-uploader-production-temp --key freetown_cog.tif --acl public-read

4. Extract Metadata

docker run --rm --entrypoint=sh ghcr.io/osgeo/gdal:alpine-small-latest

apk add jq

gdalinfo -json /vsicurl/https://oam-uploader-production-temp.s3.us-east-1.amazonaws.com/freetown_cog.tif \
| jq '{
  projection: .coordinateSystem.wkt,
  bbox: (
    .wgs84Extent.coordinates[0]
    | [.[0][0], .[0][1], .[2][0], .[2][1]]
  ),
  footprint: (
    "POLYGON ((" +
    (.wgs84Extent.coordinates[0] | map("\(. [0]) \(. [1])") | join(", ")) +
    "))"
  ),
  gsd: .geoTransform[1],
  dimensions: [.size[0], .size[1]],
  bands: (.bands | length),
  dtype: (.bands[0].type | ascii_downcase),
  crs: (
    .coordinateSystem.wkt
    | capture("ID\\[\\\"EPSG\\\",(?<code>[0-9]+)\\]")
    | "EPSG:\(.code)"
  ),
  colorinterp: [.bands[].colorInterpretation | ascii_downcase],
  resolution: [.geoTransform[1], (.geoTransform[5] | if . < 0 then -. else . end)],
  geojson: .wgs84Extent
}'

5. Get User ID

• Login to https://cloud.mongodb.com
• Find the user ID of the user you wish to be owner in the users table

6. Insert into MongoDB metas table

This is the only relevant collection we need. The images and uploads collections are simply used to track upload and processing progress.

user = 5eb37b4700afed00060e6dfe
image = 690585b76415e43597ffd7ea
meta uuid = 690585b76415e43597ffd7eb

Insert as a JSON!

{
  "_id": {
    "$oid": "690588216415e43597ffd157"
  },
  "uuid": "https://oin-hotosm-temp.s3.us-east-1.amazonaws.com/690585b76415e43597ffd7ea/0/690585b76415e43597ffd7eb.tif",
  "meta_uri": "https://oin-hotosm-temp.s3.us-east-1.amazonaws.com/690585b76415e43597ffd7ea/0/690585b76415e43597ffd7eb_meta.json",
  "user": {
    "$oid": "5eb37b4700afed00060e6dfe"
  },
  "title": "Freetown Urban with Sensitive Areas Blurred",
  "projection": "\nPROJCRS[\"WGS 84 / UTM zone 28N\",\n    BASEGEOGCRS[\"WGS 84\",\n        DATUM[\"World Geodetic System 1984\",\n            ELLIPSOID[\"WGS 84\",6378137,298.257223563,\n                LENGTHUNIT[\"metre\",1]]],\n        PRIMEM[\"Greenwich\",0,\n            ANGLEUNIT[\"degree\",0.0174532925199433]],\n        ID[\"EPSG\",4326]],\n    CONVERSION[\"UTM zone 28N\",\n        METHOD[\"Transverse Mercator\",\n            ID[\"EPSG\",9807]],\n        PARAMETER[\"Latitude of natural origin\",0,\n            ANGLEUNIT[\"degree\",0.0174532925199433],\n            ID[\"EPSG\",8801]],\n        PARAMETER[\"Longitude of natural origin\",-15,\n            ANGLEUNIT[\"degree\",0.0174532925199433],\n            ID[\"EPSG\",8802]],\n        PARAMETER[\"Scale factor at natural origin\",0.9996,\n            SCALEUNIT[\"unity\",1],\n            ID[\"EPSG\",8805]],\n        PARAMETER[\"False easting\",500000,\n            LENGTHUNIT[\"metre\",1],\n            ID[\"EPSG\",8806]],\n        PARAMETER[\"False northing\",0,\n            LENGTHUNIT[\"metre\",1],\n            ID[\"EPSG\",8807]]],\n    CS[Cartesian,2],\n        AXIS[\"(E)\",east,\n            ORDER[1],\n            LENGTHUNIT[\"metre\",1]],\n        AXIS[\"(N)\",north,\n            ORDER[2],\n            LENGTHUNIT[\"metre\",1]],\n    USAGE[\n        SCOPE[\"Navigation and medium accuracy spatial referencing.\"],\n        AREA[\"Between 18°W and 12°W, northern hemisphere between equator and 84°N, onshore and offshore. Gambia. Greenland. Guinea. Guinea-Bissau. Iceland. Ireland - offshore Porcupine Basin. Mauritania. Morocco. Senegal. Sierra Leone. Western Sahara.\"],\n        BBOX[0,-18,84,-12]],\n    ID[\"EPSG\",32628]]",
  "bbox": [
    {
      "$numberDouble": "-13.2991565"
    },
    {
      "$numberDouble": "8.5004312"
    },
    {
      "$numberDouble": "-13.1252206"
    },
    {
      "$numberDouble": "8.3971738"
    }
  ],
  "footprint": "POLYGON ((-13.2991565 8.5004312, -13.2996055 8.3979653, -13.1252206 8.3971738, -13.1247257 8.49963, -13.2991565 8.5004312))",
  "gsd": {
    "$numberDouble": "0.0394"
  },
  "file_size": {
    "$numberInt": "53193085"
  },
  "acquisition_start": {
    "$date": {
      "$numberLong": "1761105600000"
    }
  },
  "acquisition_end": {
    "$date": {
      "$numberLong": "1761109200000"
    }
  },
  "platform": "uav",
  "provider": "DroneTM",
  "contact": "Ivan Gayton,ivan.gayton@hotosm.org",
  "geojson": {
    "type": "MultiPolygon",
    "coordinates": [
      [
        [
          [
            {
              "$numberDouble": "-13.2991565"
            },
            {
              "$numberDouble": "8.5004312"
            }
          ],
          [
            {
              "$numberDouble": "-13.2996055"
            },
            {
              "$numberDouble": "8.3979653"
            }
          ],
          [
            {
              "$numberDouble": "-13.1252206"
            },
            {
              "$numberDouble": "8.3971738"
            }
          ],
          [
            {
              "$numberDouble": "-13.1247257"
            },
            {
              "$numberDouble": "8.49963"
            }
          ],
          [
            {
              "$numberDouble": "-13.2991565"
            },
            {
              "$numberDouble": "8.5004312"
            }
          ]
        ]
      ]
    ],
    "bbox": [
      {
        "$numberDouble": "-13.299349"
      },
      {
        "$numberDouble": "8.398019"
      },
      {
        "$numberDouble": "-13.124724"
      },
      {
        "$numberDouble": "8.500431"
      }
    ]
  },
  "properties": {
    "license": "CC-BY 4.0",
    "sensor": "DJI Mini 4 Pro",
    "url": "s3://oin-hotosm-temp/690585b76415e43597ffd7ea/0/690585b76415e43597ffd7eb.tif",
    "filename": "690585b76415e43597ffd7eb.tif",
    "dimensions": [
      {
        "$numberInt": "486906"
      },
      {
        "$numberInt": "287291"
      }
    ],
    "bands": {
      "$numberInt": "3"
    },
    "dtype": "uint8",
    "crs": "EPSG:32628",
    "colorinterp": [
      "red",
      "green",
      "blue"
    ],
    "resolution": [
      {
        "$numberDouble": "0.039447879950861"
      },
      {
        "$numberDouble": "0.039447979351589"
      }
    ],
    "resolution_in_meters": {
      "$numberDouble": "0.03944787995086"
    },
    "thumbnail": "https://oin-hotosm-temp.s3.us-east-1.amazonaws.com/690585b76415e43597ffd7ea/0/690585b76415e43597ffd7eb.png",
    "tms": "https://tiles.openaerialmap.org/690585b76415e43597ffd7ea/0/690585b76415e43597ffd7eb/{z}/{x}/{y}",
    "wmts": "https://tiles.openaerialmap.org/690585b76415e43597ffd7ea/0/690585b76415e43597ffd7eb/wmts",
    "tilejson": "https://tiles.openaerialmap.org/cog/tilejson.json?url=https%3A%2F%2Foin-hotosm-temp.s3.us-east-1.amazonaws.com%690585b76415e43597ffd7ea%2F0%690585b76415e43597ffd7eb.tif"
  },
  "uploaded_at": {
    "$date": {
      "$numberLong": "1761970209234"
    }
  },
  "__v": {
    "$numberInt": "0"
  }
}

Note

None of the other tables seem to be necessary.
I don't think the uploads table is needed either.

6. Ensure the COG is present in the location specified in the Mongo metadata

docker run --rm -it --entrypoint=sh docker.io/rclone/rclone:1
rclone config

rclone copy ./690585b76415e43597ffd7eb.tif hot:oin-hotosm-temp/690585b76415e43597ffd7ea/0/ --s3-no-check-bucket

7. Create a thumnail & upload to S3

gdal_translate /data/690585b76415e43597ffd7eb.tif /data/690585b76415e43597ffd7eb.png \
  -of PNG \
  -outsize 1024 0 \
  -r bilinear \
  -b 1 -b 2 -b 3 -b mask