Add `DriverManager::get_drivers_for_filename` for the ability to auto detect compatible `Driver`s for writing data #510

Atreyagaurav · 2023-12-29T19:31:50Z

I agree to follow the project's code of conduct.
I added an entry to CHANGES.md if knowledge of this change could be valuable to users.

Fixes #506

I have the tests at the end of src/driver.rs, let me know if I should remove it.

I've only tested for getting the drivers here, but I tried using that function to see if the code I wrote that wrote GPKG files can write SHP now it works. Though some driver specific functions seem to make it not work if we're not careful (like out_data.start_transaction()? works for GPKG but not SHP).

Atreyagaurav · 2024-01-01T02:23:47Z

Tested locally for:

  cargo fmt --all --check
  cargo clippy --all-targets -- -D warnings
  cargo clippy --all-targets --all-features -- -D warnings
  cargo build
  cargo test
  cargo build --all-features
  cargo test --all-features -- --nocapture

Note: The original code suggested in #506 had the tests for writing capabilities of the driver, and I've kept that as, for reading a file, Dataset::open can already read any dataset without us having to determine a driver. I've added that in docstring.

Atreyagaurav · 2024-01-01T21:04:18Z

The tests pass locally but are failing in CI, So I thought it could be because we need to install something for GPKG driver to be present in the ubuntu distribution.

So I've added conditional in the test to only test extension if the driver we're looking for is already there. It should work.

Atreyagaurav · 2024-01-02T21:04:32Z

Found the problem: The previous versions don't support .gpkg.zip and only .gpkg files.

Relevant link: https://gdal.org/drivers/vector/gpkg.html#compressed-files

$ docker run -it --entrypoint bash ghcr.io/osgeo/gdal:ubuntu-full-3.8.1
root@65b7fb97e696:/# python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from osgeo import ogr
>>> ogr.GetDriverByName("GPKG").GetMetadataItem('DMD_EXTENSIONS')
'gpkg gpkg.zip'
>>> ogr.GetDriverByName("GPKG").GetMetadataItem('DMD_EXTENSION')
>>> ogr.GetDriverByName("GPKG").GetMetadataItem('DCAP_CREATE')
'YES'
>>> ogr.GetDriverByName("GPKG").GetMetadataItem('DCAP_VECTOR')
'YES'
>>>

But

$ docker run -it --entrypoint bash ghcr.io/osgeo/gdal:ubuntu-full-3.5.3
root@6691affad24c:/# python
Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from osgeo import ogr
>>> ogr.GetDriverByName("GPKG").GetMetadataItem('DMD_EXTENSIONS')
'gpkg'
>>> ogr.GetDriverByName("GPKG").GetMetadataItem('DMD_EXTENSION')
'gpkg'

So I just made a condition on the .gpkg.zip assert in the test for newer versions only. Hopefully it works now

Atreyagaurav · 2024-01-02T21:46:48Z

I checked that all upto gdal 3.1.3 has shp.zip support:
https://gdal.org/drivers/vector/shapefile.html#compressed-files

That's why I left it as it was.

This link shows that ubuntu-lts has gdal 3.0 https://askubuntu.com/questions/1267844/installing-libgdal-dev-on-ubuntu-20-04

I've made it to check for that in the test as well.

Atreyagaurav · 2024-01-02T21:54:21Z

Ok, should I remove the ES test? I don't understand it, and I just added it for complete tests. I can't find if something changed in version 3.1 for that. Or should I add the condition of >= 3010000 for it as well.

Atreyagaurav · 2024-01-02T22:21:06Z

EDIT: Found the problem, you can only read the last 2 paragraph.

$ docker run -it --entrypoint bash ubuntu:20.04
root@1015807d8b92:/# apt install libpq-dev gdal-bin libgdal-dev
root@1015807d8b92:/# python3
Python 3.8.10 (default, Nov 22 2023, 10:22:35) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from osgeo import ogr
>>> import osgeo
>>> osgeo.__version__
'3.0.4'
>>> ogr.GetDriverByName("Elasticsearch").GetMetadataItem('DMD_CONNECTION_PREFIX')
'ES:'
>>> ogr.GetDriverByName("Elasticsearch").GetMetadataItem('DCAP_CREATE')
'YES'
>>> ogr.GetDriverByName("Elasticsearch").GetMetadataItem('DCAP_VECTOR')
'YES'
>>>

I don't know why it's failing there. the things we're checking seems fine. Should we just put gdal_version >= 3010000 for the test?

lol, the name was different ogr.GetDriverByName("Elasticsearch").name came up as ElasticSearch in 3.0.4, while for others it's Elasticsearch, even though driver_by_name gives us the driver using Elasticsearch. Should be fixed now. Hopefully no more problems.

The logic for the function is correct, tests are troublesome.

metasim · 2024-01-03T13:20:36Z

src/driver.rs

+    /// ```text
+    /// ["GPKG"]
+    /// ```
+    pub fn get_drivers_for_filename(filename: &str, is_vector: bool) -> Vec<Driver> {


Why is is_vector necessary? Is it possible to have multiple drivers that read the same file extension but differentiate on vector vs. raster data? If so, is this something we need to impose on every user (IOW, could we let the caller deal with that if it's a rare edge case?)

I guess what I'm wondering here is if we should only be evaluating against DMD_EXTENSION and DMD_EXTENSIONS, and defer capability filtering to a different mechanism (e.g. extension method on Driver). The method name implies we're only looking at matching drivers against filename (extension).

I don't know. In principle, the user might have all sorts of extra drivers installed. Matching the GDAL logic seemed like a safe approach.

We could ask Even, I guess.

The logic I went by is this: there can be multiple drivers for the same extension, and some could support raster some vector and some both. And this function is (related with other comments) made to know the driver in advance while creating a file as if we have a file already, then we can call Dataset::open without driver information. But we cannot create and save data without deciding the driver first. And if we're writing a data, we'd want the list of drivers that supports that type of data, hence that bool.

It was also in the function Inicola sent me as an example, so it's my understanding of that, not my innovation.

metasim · 2024-01-03T13:21:18Z

src/driver.rs

+            let can_create = d.metadata_item("DCAP_CREATE", "").is_some()
+                || d.metadata_item("DCAP_CREATECOPY", "").is_some();
+            let check_vector = is_vector && d.metadata_item("DCAP_VECTOR", "").is_some();
+            let check_raster = !is_vector && d.metadata_item("DCAP_RASTER", "").is_some();
+            let check_vector_translate =
+                is_vector && d.metadata_item("DCAP_VECTOR_TRANSLATE_FROM", "").is_some();


See comment above... i.e. should we consider selection and filtering two different operations?

metasim · 2024-01-03T13:23:42Z

src/driver.rs

+            let filename = filename.to_ascii_lowercase();
+            let e = match filename.rsplit_once('.') {


If we want to ensure true logical parity with GDAL C behavior, we'd use CPLGetExtension. 50/50 on whether that's good idea or not.

I did consider using the rust's "extension()" method on the Path as well, but considering that it can only do the last "." onwards, which requires us to do the endswith on the filename for .shp.zip and .gpkg.shp, I thought just using them directly might be better. And the online protocol addresses with prefix (like Elasticsearch in the test) may not be taken as a valid path, so I just did the string thing.

It would be nice to have this work for Paths. I think you can use extension and then file_stem and again extension to check for this without that much ceremony.

Path works fine with those, especially as they're just valid file names on Linux. Path::new is infallible.

I'd like this because otherwise it's pretty annoying to convert a Path or PathBuf to a string.

That's exactly I don't want to do it lol.

Like for that: startswith() we need to do for "PG:" and similar paths, we'd need to convert the path into string right? And iirc it can't convert it into direct string and has to go through Debug trait and ends up with quoted string or something.

I can try, but unless we can use the Path args without having to turn it back to string, it feels like we'll just end up asking for a Path and then internally convert it into string anyway.

if you have a Path it's harder to get a string.

That conflicts with the first sentence :D

Users can just call [to_string_lossy](https://doc.rust-lang.org/std/path/struct.Path.html#method.to_string_lossy). If it's easy for us, it's easy for them.

Anyway, I'm trying to do it with AsRef<Path>. I'll make a commit if I am successful.

Edit: users have to call it once to pass the path as string currently.

If we take Path, we have to call to_string_lossy for extension, for filename (second extension), and again to check for prefix...

If it's easy for us, it's easy for them.

It's easy for us in the sense that it's infallible, it requires no unsafe code, and it's almost always efficient. It's harder for them because it's ugly and maybe they won't know about to_string_lossy so they might try to debug-format them, as you wanted to do.

There's also something to be said about being lossy, but I guess we only support UTF-8 file names, while GDAL might, in principle, support arbitrary byte sequences). But we hide it, and we could support non-UTF-8 file names today -- at least on Linux -- without changing our API.

On the other hand, if we make the user call to_string_lossy on a non-UTF-8 path, they'll get a different name at the output. It's fine when we do it (and also when the user does it) because it won't affect the driver selection in practice. But as you can see, it's a lot of relatively subtle stuff to think about.

Also, you'll see that to_string_lossy returns a Cow<'_, str>, which might cause some people to give up and go back to Python.

Haha, I'm currently having that same problem. it seems I can't use Cow<'_, str> on pattern. And might have to do, Variants of Cow::. At this point doing if statement seems easier than match. It's really a lot of complications when we have to turn it back into string. :(

Okay, let's leave it like this, I'll try to clean it up tomorrow.

Because of the abovementioned problem with Cow, and not being able to use match properly and so on. I converted it into String after fighting with borrow checking for a while trying to use str.

metasim · 2024-01-03T13:33:21Z

src/driver.rs

+        for i in 0..DriverManager::count() {
+            let d = DriverManager::get_driver(i).expect("Index for this loop should be valid");
+            let can_create = d.metadata_item("DCAP_CREATE", "").is_some()
+                || d.metadata_item("DCAP_CREATECOPY", "").is_some();
+            let check_vector = is_vector && d.metadata_item("DCAP_VECTOR", "").is_some();
+            let check_raster = !is_vector && d.metadata_item("DCAP_RASTER", "").is_some();
+            let check_vector_translate =
+                is_vector && d.metadata_item("DCAP_VECTOR_TRANSLATE_FROM", "").is_some();


See this description of what GDAL does when Dataset::open is called.
https://github.com/OSGeo/gdal/blob/master/gcore/gdaldataset.cpp#L3630-L3645

From my 5 minute reading it looks like some of the logic is delegated to GDALIdentifyDriverEx:

https://github.com/OSGeo/gdal/blob/master/gcore/gdaldriver.cpp#L2649C25-L2649C45

My concern here is not wanting to diverge from the accepted GDAL logic.

GDALIdentifyDriverEx is used for opening, but that's usually not necessary.

The functions I linked to in #506 (comment) is used by the GDAL binaries to pick an output format. If we offer this function, we should match the latter, and use a better name (get_driver(s)_for_writing or something like that).

We do need to change the function name. I was also looking for ideas as the ones I came up got too long.

The functions I linked to in #506 (comment) is used by the GDAL binaries to pick an output format.

Ahhh... I thought there was something already in the code, but didn't realize it was C++ only :/

Yeah, it is for binary apps. Not the part of the gdal library. Otherwise, I was planning on just using the function from gdal-sys.

src/driver.rs

metasim · 2024-01-03T13:36:44Z

src/driver.rs

+    /// drivers and returns the matches. The determined driver is
+    /// checked for writing capabilities as
+    /// [`Dataset::open`](Dataset::open) can already open datasets
+    /// with just path.


Not sure I understand why we limit returned drivers to ones with "write" capabilities... maybe read-only use cases might also want a list of drivers to pick from (e.g. when reading JP2k, there are several options and you might want to inspect the metadata to decide on which to select).

So, initially I was thinking that, but then I realized that when we already have a dataset, we can just call Dataset::open and then look at the driver. Having a dataset you can open means you have a lot in your arsenal to detect the driver, like the metadata you mentioned.

The use case we're thinking is to when we need to write a dataset, currently the only way I could find to write to file is to create a file from a driver, which requires you to know the driver in advance. Of course, you can make the user pass the driver name along with the filename and call it a day (that's what I was doing on one of my program), but all users are not likely to know about the driver names. And if we can more or less guess based on the file extension, then we make it convenient and remove any user error (could write a Tiff file in .shp extension file and it'll write it). And since GIS programs let you write to any file, making the program only dependent on the filename makes it easier to integrate with other programs as well.

I think renaming the function to mention it's for writing should help with it.

metasim · 2024-01-03T13:38:05Z

I have the tests at the end of src/driver.rs, let me know if I should remove it.

That's the appropriate place :)

Atreyagaurav · 2024-01-03T14:47:01Z

That's the appropriate place :)

Thank you. That was more of a comment due to worry that the tests might fail on the CI or on other people's computer because some drivers are not available. But that should be fixed now. Thought the tests did give a lot of trouble for other reasons.

lnicola · 2024-01-03T14:48:36Z

Nit: maybe we should use something more common like PG instead of ElasticSearch?

Atreyagaurav · 2024-01-03T14:54:52Z

Nit: maybe we should use something more common like PG instead of ElasticSearch?

Done.

I also renamed the funtion. Idk if PG will also have some quirks on the tests like ElasticSearch though. Hopefully not

src/driver.rs

metasim · 2024-01-03T16:47:41Z

src/driver.rs

+    /// ```text
+    /// ["GPKG"]
+    /// ```
+    pub fn guess_drivers_for_write(filename: &str, is_vector: bool) -> Vec<Driver> {


Sorry to bikeshed here, but should we consider:

Also providing guess_drivers_for_read

Have the signature guess_drivers_for_write<P: AsRef<Path>>(filename: P, ...) to be more coherent with other methods accepting files. (&str is AsRef<Path>).

Not sure about guess_drivers_for_read. There's already an API that works for existing files, and do we really need to find a driver for a file that doesn't exist?

I did consider using AsRef<Path>, but I thought the connections like PG:https://xxxxxx might not work or have unexpected problems. And we need to convert it back to string to use the "endswith" or "split" for double extensions. Hence, I went with string. I haven't tested if those connections work as path or not though.

metasim · 2024-01-03T18:17:33Z

LGTM

metasim · 2024-01-09T01:26:44Z

Also, if we want to wait to use the gdal-sys function from the new version, we can wait. Otherwise, we can merge this, and then later replace the implementation by calling that function internally.

This is my vote, assuming the semantics are/will be the same. Would be nice to expose the capability for users using older versions of 3.x, but then add a cfg block for the new version of gdal, upon release.

lnicola · 2024-01-09T19:42:10Z

Yeah, I think we can add it now (with an appropriate signature for the native one), with a note that it's implemented differently before 3.9.

lnicola · 2024-01-10T19:23:38Z

I pushed a couple of changes, but still want to do another pass before merging, sorry.

Atreyagaurav · 2024-01-10T19:58:36Z

Wouldn't the name get_drivers_for_path be better than name? As name is used as driver name driver_by_name, so it might be ambiguous. So filename or path might be better.

lnicola · 2024-01-10T20:01:12Z

Yeah, probably. I had dataset_name, but it seemed a little excessive. path is confusing for PG.

Atreyagaurav · 2024-01-10T20:08:25Z

It's still a path in case of PG and others, just not a local one. Should be fine if people check the doc strings, if they don't I thought name might make people try the driver name instead.

should help implementing georust/gdal#510

Atreyagaurav · 2024-01-15T17:39:43Z

This is finalized, right? No more modifications?

metasim · 2024-01-15T17:52:50Z

CHANGES.md

@@ -10,6 +10,8 @@

  - <https://github.com/georust/gdal/pull/504>

+- Added `DriverManager::guess_driver_for_write` and `DriverManager::guess_drivers_for_write` for the ability to auto detect compatible `Driver`(s) for writing data.
+


Don't forget to add the PR link here.

Thank you. Done.

metasim · 2024-01-15T17:54:32Z

src/driver.rs

@@ -461,6 +592,11 @@ impl DriverManager {
    }
 }

+pub enum DriverProperties {


Consider the name DriverType, as "Properties" usually implies a collection of settings.

Thank you, I was initially thinking that properties might be weird. It's renamed now.

metasim · 2024-01-15T18:02:29Z

Looks good to me. @lnicola OK if I merge?

lnicola · 2024-01-15T18:38:36Z

Yes, but I think it still needs a rebase.

I'm a little worried about that enum, since we should probably add a variat (bit) to warn when there are multiple matching drivers (GDAL has a second parameter, but we don't need to match that). That would make the name a little unfortunate, and something like DriverPreference or GetOutputDriverFlags would be nicer.

But let's merge this now, as I'm not going to have much time for it in the near future and I don't want to keep delaying it forever.

metasim · 2024-01-15T18:41:55Z

Is a "Squash and merge" sufficient?

lnicola · 2024-01-15T19:07:27Z

Feel free to try it, I'm seeing:

This branch cannot be rebased due to conflicts

metasim · 2024-01-15T19:15:24Z

@Atreyagaurav thanks for your hard, patient work on this. We all appreciate it! :)

Atreyagaurav · 2024-01-15T19:28:25Z

Thank you everyone for reviewing and the guidance. And for making gdal available in rust. I'll try to contribute more if I find some simple issues I can fix.

Atreyagaurav added 4 commits December 28, 2023 18:14

Add DriverManager::get_drivers_for_filename

5f44a24

Add docstring to DriverManager::get_drivers_for_filename; use match

9fb996b

Satisfy clippy

f7d6bcb

Improve readability for DriverManager::get_drivers_for_filename

3d569e0

only test for available drivers in DriverManager::get_driver_by_name

0b8d80d

Atreyagaurav added 2 commits January 1, 2024 16:07

Add test for ESRI Shapefile for .shp file extension

79300cd

bool for vector/raster in DriverManager::get_drivers_for_filename

563eb97

Atreyagaurav force-pushed the master branch from 6ff1fbf to 563eb97 Compare January 1, 2024 23:50

Fix: modify test for gpkg.zip only for gdal v 3.7 onwards

62f7ccd

Fix: modify test for shp.zip only for gdal v 3.1 onwards

3dbc42f

Atreyagaurav force-pushed the master branch from 12ebe8a to 3dbc42f Compare January 2, 2024 21:46

Fixed the test failed due to Elasticsearch name capitalization change

05f46c4

metasim requested changes Jan 3, 2024

View reviewed changes

lnicola reviewed Jan 3, 2024

View reviewed changes

src/driver.rs Outdated Show resolved Hide resolved

Atreyagaurav force-pushed the master branch from 0b6c125 to 83d60c6 Compare January 3, 2024 14:59

metasim reviewed Jan 3, 2024

View reviewed changes

metasim approved these changes Jan 3, 2024

View reviewed changes

Rename the function and minor changes

7c90f62

Atreyagaurav force-pushed the master branch from 83d60c6 to 7c90f62 Compare January 3, 2024 18:27

Atreyagaurav and others added 2 commits January 9, 2024 15:46

Use AsRef<Path> instead of &str in guess_driver(s)_for_write

45473a6

Small cleanups

fc0361a

lnicola mentioned this pull request Jan 10, 2024

Random test suite crash #514

Open

lnicola added 2 commits January 10, 2024 21:26

Fix test

2c1fc76

Try to debug test

773872d

lnicola force-pushed the master branch from a8ce4d1 to 773872d Compare January 10, 2024 19:35

Remove debugging code

a187d38

rouault added a commit to rouault/gdal that referenced this pull request Jan 10, 2024

Add GDALGetOutputDriversForDatasetName()

c8bb107

should help implementing georust/gdal#510

rouault added a commit to rouault/gdal that referenced this pull request Jan 11, 2024

Add GDALGetOutputDriversForDatasetName()

e44f142

should help implementing georust/gdal#510

Rename methods and ignore case

f54cd81

metasim reviewed Jan 15, 2024

View reviewed changes

Add PR link to CHANGES.md

326feda

metasim reviewed Jan 15, 2024

View reviewed changes

Atreyagaurav added 2 commits January 15, 2024 12:56

Rename DriverProperties to DriverType

04a549c

Fix: wrong PR link location

98fbc23

Atreyagaurav force-pushed the master branch from d8b46a5 to 98fbc23 Compare January 15, 2024 18:02

metasim merged commit 87497bf into georust:master Jan 15, 2024
9 checks passed

		let filename = filename.to_ascii_lowercase();
		let e = match filename.rsplit_once('.') {

		@@ -10,6 +10,8 @@

		- <https://github.com/georust/gdal/pull/504>

		- Added `DriverManager::guess_driver_for_write` and `DriverManager::guess_drivers_for_write` for the ability to auto detect compatible `Driver`(s) for writing data.

Add DriverManager::get_drivers_for_filename for the ability to auto detect compatible Drivers for writing data #510

Add DriverManager::get_drivers_for_filename for the ability to auto detect compatible Drivers for writing data #510

Conversation

Atreyagaurav commented Dec 29, 2023

Atreyagaurav commented Jan 1, 2024

Atreyagaurav commented Jan 1, 2024

Atreyagaurav commented Jan 2, 2024

Atreyagaurav commented Jan 2, 2024 • edited

Atreyagaurav commented Jan 2, 2024

Atreyagaurav commented Jan 2, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Atreyagaurav Jan 9, 2024 • edited

Choose a reason for hiding this comment

Anyway, I'm trying to do it with AsRef<Path>. I'll make a commit if I am successful.

lnicola Jan 9, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

metasim commented Jan 3, 2024

Atreyagaurav commented Jan 3, 2024

lnicola commented Jan 3, 2024

Atreyagaurav commented Jan 3, 2024

Choose a reason for hiding this comment

lnicola Jan 3, 2024 • edited

Choose a reason for hiding this comment

Atreyagaurav Jan 3, 2024 • edited

Choose a reason for hiding this comment

metasim commented Jan 3, 2024

metasim commented Jan 9, 2024

lnicola commented Jan 9, 2024 • edited

lnicola commented Jan 10, 2024

Atreyagaurav commented Jan 10, 2024

lnicola commented Jan 10, 2024

Atreyagaurav commented Jan 10, 2024

Atreyagaurav commented Jan 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

metasim commented Jan 15, 2024

lnicola commented Jan 15, 2024

metasim commented Jan 15, 2024

lnicola commented Jan 15, 2024

metasim commented Jan 15, 2024

Atreyagaurav commented Jan 15, 2024

Add `DriverManager::get_drivers_for_filename` for the ability to auto detect compatible `Driver`s for writing data #510

Add `DriverManager::get_drivers_for_filename` for the ability to auto detect compatible `Driver`s for writing data #510

Atreyagaurav commented Jan 2, 2024 •

edited

Atreyagaurav commented Jan 2, 2024 •

edited

Atreyagaurav Jan 9, 2024 •

edited

Anyway, I'm trying to do it with `AsRef<Path>`. I'll make a commit if I am successful.

lnicola Jan 9, 2024 •

edited

lnicola Jan 3, 2024 •

edited

Atreyagaurav Jan 3, 2024 •

edited

lnicola commented Jan 9, 2024 •

edited