Support opening vector datasets with GDALOpenEx

[callmeahab]

I'd like to know if there are any plans to support reading/writing GDB files?
Also are there some limitations that would prevent implementing this functionality?

[ddohler]

Loam relies on gdal-js, which is a version of GDAL compiled to WebAssembly, so it can (in theory) read/write any file formats that GDAL can without requiring external libraries, as of version 2.1.

It looks like GDAL has built-in support for read-only GDB access, for vector data only, but not raster data. Does that satisfy your use-case?

There is also a read-write GDB driver available for GDAL, but it requires an external library, so it would be necessary to create a custom build of gdal-js in order to support it. This would be some work, but not impossible.

If you need raster data, it seems like you're out of luck, unfortunately -- GDAL doesn't support reading or writing raster data in GDB files.

I hope that helps!

[callmeahab]

Thanks for the quick response. Yeah I'm looking for read-only vector data with GDB files. Would this question be more appropriate for gdal-js repo in that case?

[callmeahab]

Just one more question, in case I do manage to build gdal-js with GDB support, will I need to update something in this repo to support it or will things work "out of the box" so to speak?

[ddohler]

Yeah I'm looking for read-only vector data with GDB files. Would this question be more appropriate for gdal-js repo in that case?

If all you're looking for is read-only vector data, I think OpenFileGDB driver should already be compiled into gdal-js and so it should just work as-is. However, I don't know of anyone who has tried it, so it's certainly possible that it doesn't work. If you've tried it already and encountered problems, I'd be happy to take a look at whatever errors you're seeing!

If you haven't tried using a GDB file yet, an easy way to test would be to clone the repo, check out the develop branch and try using a GDB file on the demo page (this functionality will be merged to master in the next release, which I'm hoping to get to next week).

Just one more question, in case I do manage to build gdal-js with GDB support, will I need to update something in this repo to support it or will things work "out of the box" so to speak?

You might need to pass the location of your custom gdal-js build to loam.initialize(), but otherwise it should work as-is (but again, no one has tried this to my knowledge, so if you run into problems, I'd be interested in hearing about them).

In general most of the Loam functionality uses GDAL API functions pretty directly, so if you would expect something to work using gdal_translate or similar, then I would expect it to work with Loam, assuming GDAL had the necessary file support compiled in. If the GDAL file support was there but something didn't work with Loam then I'd most likely consider that a bug in Loam.

[callmeahab]

Ok, I did try using a zipped gdb archive and encountered this error:

Unhandled Rejection (Error): Error in GDALOpen: `/1xrr4jeqf1r2hw31q3mm5dd7qz9uti3a/Aerial Pilot.gdb.zip' not recognized as a supported file format.

Worker._worker.onmessage
/src/workerCommunication.js:36
  33 |     if (msg.data.success) {
  34 |         messages[msg.data.id][0](msg.data.result);
  35 |     } else {
> 36 |         messages[msg.data.id][1](new Error(msg.data.message));
     | ^  37 |     }
  38 |     delete messages[msg.data.id];
  39 | };

Thanks again for responsiveness.

[callmeahab]

I also tried .shp.zip file and got the same error, same for a geojson file.

[ddohler]

Oh, I see what's going on -- Loam wraps the GDAL* parts of the library, but there are no wrappers for OGR yet.

I would expect Loam to exhibit the same behavior as gdalinfo with respect to supported libraries, and indeed:

$ gdalinfo map.geojson 
ERROR 4: `map.geojson' not recognized as a supported file format.
gdalinfo failed - unable to open 'map.geojson'.

But ogrinfo works:

$ ogrinfo map.geojson
INFO: Open of `map.geojson'
      using driver `GeoJSON' successful.
1: map (Multi Polygon)

This is the second request that I've had for wrappers for the OGR API, so that makes me want to prioritize it. Could you give me some more information about what you would want to do with the dataset once it has been opened? Wrapping OGROpen on its own doesn't do much so I'd like to know what other API methods to wrap along with it to provide something useful.

Secondly, I think that either way, a .zip file won't work -- if I try ogrinfo on a .shp.zip it doesn't work -- are you seeing different behavior?

[callmeahab]

I want to open a gdal file get field definitions, iterate through records, render on a map. I'm not sure about other tools but opening the file from C API works well with zipped gdb or shp files.

Just want to clarify, will writing my own worker for gdal-js work for my use case. Or if you want I could take a look into implementing wrapper in loam (although I'll need some guidance for that).

Thanks

[ddohler]

I'm not sure about other tools but opening the file from C API works well with zipped gdb or shp files.

Hmm, interesting. Which C API function are you using to do that, and what version of GDAL are you using?

Just want to clarify, will writing my own worker for gdal-js work for my use case. Or if you want I could take a look into implementing wrapper in loam (although I'll need some guidance for that).

If you're comfortable writing C and you only need to do that one thing, then the gdal-js path might be easiest. You'd need to:

1. Edit https://github.com/ddohler/gdal-js/blob/develop-js/Makefile#L66 to add the C API functions you want to expose.
2. Recompile gdal-js following the instructions here
3. Take a look at one of the gdal-js examples for how to use gdal-js; it's really quite similar to writing C.
4. Make a PR to gdal-js exposing the new functions (if you want); I'm always happy to expose more functions there because they don't increase the bundle size much, and then I can write the Loam wrapper later.

[callmeahab]

Hmm, interesting. Which C API function are you using to do that, and what version of GDAL are you using?

I'm using GDALOpenEx just like in their tutorial here: https://gdal.org/tutorials/vector_api_tut.html, and using GDAL 3.x. Zip file support might be new in version 3.

Thanks for the instructions, I'll check out how to add those functions.

[ddohler]

Ah, okay, you might not need to add any new functions to gdal-js then because GDALOpenEx is already exposed. That being the case, I'm going to change the title of this issue to "Support Vector datasets with GDALOpenEx", since I suspect the main change that would be necessary to Loam would be switching to GDALOpenEx rather than GDALOpen for opening datasets, given that that already works for you with the C API.

[callmeahab]

I'll create a PR with some additional functions needed for working with vector features. Can you give me a permission to push to a branch?

[ddohler]

I'd suggest creating a fork -- then you'll be able to push to a branch on the forked repo and make a PR to the upstream repo from your forked copy. Looking forward to seeing it!