Use bindgen for gdal-sys

[lnicola]

Closes #50
Closes #56

TODO:

• figure out how to find the GDAL include path
• clean up bindgen config
• choose between libc and std::os::raw
• see if it makes sense to split the bindings into sub-modules
• decide whether to use the modules for enums
• use modules for the remaining enums
• pick again between 1 and TRUE
• clean up the pointer casts
• bring back re-exports in submodules
• see where the generated file should live
• figure out the Windows story
• decide whether to use bindgen or commit the generated file
• decide whether to use the system GDAL headers
• figure out why my headers are missing parameter names
• try to use the generated handle types instead of *mut c_void
• consider replacing WkbType with OGRwkbGeometryType
• decide whether we really want to do this
• do something about OGRErr
• look into i32 vs. u32 for enums
• allow setting the include and link paths
• allow static GDAL builds
• update gdal-sys build instructions
• bump version

[jdroenner]

This is really great work :)

It was not easy to get the crate building on Windows with mingw as well as the MS tool-chain. So please make sure that it still builds on Windows. If you don't have a Windows system around, just tell me and i will test it.

[lnicola]

Thanks, but I'm pretty sure it's no longer working. Unfortunately, I don't have a way to test it, neither on MSVC, nor in MinGW. So it would be good if you could look into it.

One weird thing is that I no longer need a cargo:rustc-link-lib declaration (cargo outputs a warning). I expect the same is true on MinGW. The pkg-config part might work there, but it won't know about GDAL_HOME, so it will probably be a no-op.

[jdroenner]

No problem. I will test it later today :)

[jdroenner]

Building works but the tests won't run. Will try to fix it tomorrow....

[jdroenner]

It appears to be a problem with my machine or windows in general. The test are not running on master, too :(

[lnicola]

Are you getting any error, either as a message box or in Event Viewer? Maybe it can't find the GDAL DLL?

In the meanwhile, do you have any thoughts about the remaining TODO items?

[jdroenner]

see if it makes sense to split the bindings into sub-modules

I don't know if it makes sense to split the bindings. What modules would that be? Since GDAL 2.x OGR and GDAL are merging...

decide whether to use bindgen or commit the generated file

This is a bit more complicated. I guess we can do a new release for new GDAL releases and running bindgen is not really required for each build...

consider replacing WkbType with OGRwkbGeometryType

This would be the way to go. I think we already use all other enums.

decide whether we really want to do this

we will never be able to cover all of GDAL if we create the bindings by hand

[lnicola]

I don't know if it makes sense to split the bindings. What modules would that be? Since GDAL 2.x OGR and GDAL are merging...

I didn't realize this.

I guess we can do a new release for new GDAL releases and running bindgen is not really required for each build...

Or we can do both with a feature. But I'm worried abut compatibility. Will old binding work with a newer GDAL? Will new bindings work with an older GDAL as long as the application isn't calling any missing method? In the latter case, won't the OS loader complain, at least on Windows?

I looked at libsqlite3-sys, and it does this: https://github.com/jgallagher/rusqlite/blob/master/libsqlite3-sys/Cargo.toml#L13.

[jdroenner]

Maybe we should go with the features approach. PyGDAL has a binding for every release but i'm not yet convinced that this is required. GDAL releases are very conservative in my experience.

[jdroenner]

surprisingly it works with the gnu toolchain on windows on a different machine.
It requires installing clang (that was expected) and adding .clang_arg("-IC:\\OSGeo4W64\\include") to the bindgen builder.

The MSVC toolchain does not work but i think i can figure it out with a bit more time.
C:\OSGeo4W64\include/cpl_port.h:137:10: fatal error: 'stdio.h' file not found

[lnicola]

surprisingly it works with the gnu toolchain on windows on a different machine.
It requires installing clang (that was expected) and adding .clang_arg("-IC:\OSGeo4W64\include") to the bindgen builder.

Maybe we could bundle at least some bindings, so people don't have to deal with that?

The MSVC toolchain does not work but i think i can figure it out with a bit more time.

Ouch. So clang is not finding the VS include path.

[jdroenner]

I just reinstalled the VS build tools and now it also works with the MSVC tool chain 😄
But this time only the bindgen version is working... Both, MSVC and GNU toolchains won't build with the pre-generated bindings.

C:...\target\debug\deps\gdal-b8b9d3d1e4a9112e.1d37qlmqawo0lpmo.rcgu.o: In function gdal::metadata::Metadata::description::h12a3509c01c93404:
C:.../src/metadata.rs:11: undefined reference to GDALGetDescription

Which version of GDAL did you use to generate the bindings?

[jdroenner]

Uuh this was really obvious...

You have to run the #[cfg(windows)] part regardless if you are using bindgen or use pre-generated bindings.
We should move it into a special method prepare_windows and call it first in the main method #[cfg(windows)] prepare_windows();.

[lnicola]

I've been trying to make this work for multiple versions of GDAL, but I ran into rust-lang/rust-bindgen#1267.

[jdroenner]

great! could you add the configurable include paths? I guess this will also allow to build on windows.

[lnicola]

Yes, I'll try to look into it tomorrow.

[lnicola]

@jdroenner, @weiznich, can you take a look at 9bde2eb?

[jdroenner]

i will try it later today when i have access to a windows pc.

[jdroenner]

all i get is "error: could not find native static library gdal_i.lib, perhaps an -L flag is missing?"

[lnicola]

Can you add a print or panic! here to see what path it picks?

[lnicola]

Ah, I think it uses gdal_i.lib instead of gdal_i.

[jdroenner]

Its working now 👍 Great work!

[lnicola]

Okay, so I've gone through my TODO list.

[jdroenner]

This is really great 👍 Thanks a lot!
I guess we need to release a new version of rust-gdal as well :)

[lnicola]

As a test, I tried to port https://github.com/t-rex-tileserver/t-rex to the new version. It was trivial, but I had to add a reference to gdal-sys because gdal_sys::OGRwkbGeometryType isn't re-exported.

Should we re-export these? It might be nicer for the users.

[lnicola]

Oh, it actually is. Please ignore my last comment.

[lnicola]

The pre-built files can be obtained with:

bindgen --constified-enum-module ".*" --ctypes-prefix libc --whitelist-function "(CPL|GDAL|OGR|OSR|OCT|VSI).*" wrapper.h

For some reason, that won't generate the docstrings. In practice, I used something like:

bindgen --constified-enum-module ".*" --ctypes-prefix libc --whitelist-function "(CPL|GDAL|OGR|OSR|OCT|VSI).*" wrapper.h -- -I ~/Downloads/gdal/gdal-2.3/usr/include >! prebuilt-bindings/gdal_2.3.rs

See rust-lang/rust-bindgen#1265.