Support MacOS #33

clemens-tolboom · 2021-01-24T17:57:04Z

Trying to compile on MacOS I still fail due to lack of C++ routine. Help welcome

Prepare

checkout PR

git clone git@github.com:boku-ilen/geodot-plugin.git
cd geodot-plugin
git fetch origin pull/33/head:feature/support-macos
git checkout feature/support-macos

Install GDal using homebrew

brew install gdal

Get path of GDAL install

brew info gdal | grep '/usr/local' | cut -d ' ' -f 1

'normal' compile prepare steps

git submodule update --init --recursive

Build GDNative support

cd godot-cpp
time scons generate_bindings=yes --jobs=$(sysctl -n hw.logicalcpu)
cd ..

Build final code

Run scons using osgeo_path and either debug or release
scons target=debug osgeo_path=/usr/local/Cellar/gdal/3.2.1
scons target=release osgeo_path=/usr/local/Cellar/gdal/3.2.1

Check the result as addons must be filled

git status
find demo/addons/geodot/osx/

Run test scene

Assume you have shell access just type

cd demo
godot GeopackageDemo.tscn

Cleanup

To test rebuilding remove all files generated.

git clean -dfX
git reset --hard
git clean -fd demo/

clemens-tolboom · 2021-01-24T17:58:46Z

Running

scons --platform=osx

results in

scons: Reading SConscript files ...
note: scons_compiledb could not be imported. To be able to use the `--compiledb=` flag, please run `pip3 install scons-compiledb`.
scons: Reading SConscript files ...
OSGeo paths are invalid!
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
g++ -o Feature.os -c -fPIC -I/usr/local/Cellar/gdal/3.2.1/include -I. -I/Users/clemens/Downloads/godot/geodot-plugin/src/global Feature.cpp
In file included from Feature.cpp:1:
./Feature.h:18:26: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
    virtual ~Feature() = default;
                         ^
./Feature.h:32:32: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
    GeometryType geometry_type = NONE;
                               ^
Feature.cpp:14:10: warning: 'auto' type specifier is a C++11 extension [-Wc++11-extensions]
    for (auto &&oField : *feature) {
         ^
Feature.cpp:14:15: warning: rvalue references are a C++11 extension [-Wc++11-extensions]
    for (auto &&oField : *feature) {
              ^
Feature.cpp:14:24: warning: range-based for loop is a C++11 extension [-Wc++11-extensions]
    for (auto &&oField : *feature) {
                       ^
Feature.cpp:14:26: error: cannot use incomplete type 'OGRFeature' as a range
    for (auto &&oField : *feature) {
                         ^
./Feature.h:8:7: note: forward declaration of 'OGRFeature'
class OGRFeature;
      ^
Feature.cpp:22:19: error: member access into incomplete type 'OGRFeature'
    return feature->GetFieldAsString(name);
                  ^
./Feature.h:8:7: note: forward declaration of 'OGRFeature'
class OGRFeature;
      ^
5 warnings and 2 errors generated.
scons: *** [Feature.os] Error 1
scons: building terminated because of errors.
FileNotFoundError: [Errno 2] No such file or directory: 'src/raster-tile-extractor/build/libRasterTileExtractor.dylib':
  File "/Users/clemens/Downloads/godot/geodot-plugin/SConstruct", line 168:
    copyfile(os.path.join(rte_libpath, rte_library) + lib_file_ending, os.path.join(env['target_path'], rte_library) + lib_file_ending)
  File "/Users/clemens/lib/miniconda3/lib/python3.7/shutil.py", line 120:
    with open(src, 'rb') as fsrc:

kb173 · 2021-01-24T21:17:42Z

Thanks again for your work on this! In addition to defines.h, you'll need to add an #elif __APPLE__ branch to the GDAL includes in these files:

GeoRaster.cpp
RasterTileExtractor.cpp
Feature.cpp
LineFeature.cpp
PointFeature.cpp
PolygonFeature.cpp
VectorExtractor.cpp

Unfortunately I don't know how the #include should look on Mac, but I'm guessing it might be the same as on Linux. Could you try that and report back?

Once that is done, it should compile, unless there's hidden OS-specific code which only causes problems on Mac, not between Windows and Linux.

clemens-tolboom · 2021-01-25T10:36:16Z

With currrent PR I can build some code. But src/raster-tile-extractor/build/libRasterTileExtractor.dylib is not build.

scons --platform=osx --osgeo_path=/usr/local/Cellar/gdal/3.2.1

gives

scons: Reading SConscript files ...
note: scons_compiledb could not be imported. To be able to use the `--compiledb=` flag, please run `pip3 install scons-compiledb`.
scons: Reading SConscript files ...
OSGeo paths are invalid!
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.

of which

FileNotFoundError: [Errno 2] No such file or directory: 'src/raster-tile-extractor/build/libRasterTileExtractor.dylib':

puzzles me

  File "/Users/clemens/Downloads/godot/geodot-plugin/SConstruct", line 168:
    copyfile(os.path.join(rte_libpath, rte_library) + lib_file_ending, os.path.join(env['target_path'], rte_library) + lib_file_ending)
  File "/Users/clemens/lib/miniconda3/lib/python3.7/shutil.py", line 120:
    with open(src, 'rb') as fsrc:

clemens-tolboom · 2021-01-25T11:00:20Z

I can now compile.

Why are both SConstruct files different for windows? #38 Why are both SConstruct files different for windows?
diff src/vector-extractor/SConstruct src/raster-tile-extractor/SConstruct
The linux includes all <gdal/... ... can't we make this uniform? #39 The linux includes all <gdal/... ... can't we make this uniform?
grep --recursive --after=2 __unix__ .
What is env.Append(LINKFLAGS=['/WHOLEARCHIVE']) and should osx have that too? #40 What is env.Append(LINKFLAGS=['/WHOLEARCHIVE']) and should osx have that too?
More?

clemens-tolboom · 2021-01-25T11:26:45Z

Following the steps above I see one warning

g++ -o VectorExtractor.os -c -std=c++17 -fPIC -I/usr/local/Cellar/gdal/3.2.1/include -I. -I/Users/clemens/Downloads/godot/geodot-plugin/src/global VectorExtractor.cpp
VectorExtractor.cpp:208:5: warning: ignoring return value of function declared with 'warn_unused_result' attribute [-Wunused-result]
    square_layer->CreateFeature(square_feature);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
1 warning generated.

and 1 warning which should be a fat error

g++ -o demo/addons/geodot/osx/libgeodot.dylib -arch x86_64 -dynamiclib src/gdlibrary.os src/geodata.os src/geodot.os src/geofeatures.os src/geoimage.os -Lgodot-cpp/bin -Lsrc/raster-tile-extractor/build -Lsrc/vector-extractor/build -Llib -lgodot-cpp.osx.debug.64 -lRasterTileExtractor -lVectorExtractor -lgdal
ld: warning: directory not found for option '-Llib'

clemens-tolboom · 2021-01-25T11:30:21Z

Running scons target=d osgeo_path=/usr/local/Cellar/gdal/3.2.1 solved building.

Godot

ERROR: open_dynamic_library:
  Can't open dynamic library:
    /Users/clemens/Downloads/godot/geodot-plugin/demo/addons/geodot/osx/libgeodot.dylib
  , error: dlopen(/Users/clemens/Downloads/godot/geodot-plugin/demo/addons/geodot/osx/libgeodot.dylib, 2):
 Library not loaded: build/libRasterTileExtractor.dylib
  Referenced from: /Users/clemens/Downloads/godot/geodot-plugin/demo/addons/geodot/osx/libgeodot.dylib
  Reason: image not found.
   At: platform/osx/os_osx.mm:1937.
ERROR: get_symbol: No valid library handle, can't get symbol from GDNative object
   At: modules/gdnative/gdnative.cpp:501.
ERROR: init_library: No nativescript_init in "res://addons/geodot/osx/libgeodot.dylib" found
   At: modules/gdnative/nativescript/nativescript.cpp:1506.
ERROR: _create_autoload: Script does not inherit a Node: res://addons/geodot/geodot.gdns.
   At: editor/editor_autoload_settings.cpp:357.

kb173 · 2021-01-25T11:34:50Z

Very nice! Regarding the Windows builds and env.Append(LINKFLAGS=['/WHOLEARCHIVE']), I'll ping @MathiasBaumgartinger - can you try removing that line (line 42) from the SConstruct script in raster-tile-extractor and check if it still compiles on Windows?

The whole Windows build pipeline was a result of a lot of trial and error, so it's possible that the script includes things that we ended up not actually needing.

Regarding the second point, yeah, we could put all those GDAL includes into one file which we include everywhere else. That way we have all the ugly OS guards in one place.

Is the call producing ld: warning: directory not found for option '-Llib' generated by SCons? If so, could it be because of lines 151 to 153 in the SConstruct script? These:

elif env['platform'] == 'osx':
    env.Append(LIBS=['libgdal.dylib'])
    env.Append(LIBPATH=[os.path.join(env['osgeo_path'], "lib")])

I think the lines above were a pretty Windows-specific fix, and the "lib" parameter seems like it could produce the option '-Llib' mentioned in the error.

Regarding your last comment: Does build/libRasterTileExtractor.dylib exist, and did it build without warnings?

kb173 · 2021-01-25T11:39:08Z

Just a guess: Could it be that it's expecting libRasterTileExtractor.dylib to be in a directory called build for some reason, but it's actually in the same directory as libgeodot.dylib?

MathiasBaumgartinger · 2021-01-25T11:51:25Z

@kb173 Does compile without that line for me too! An important note however is that for me to work i need to build for release or i end up having lots of errors. Done via an extra CL-arg: target=release

clemens-tolboom · 2021-01-25T13:35:02Z

I guess I should have created new issues for my remarks ... not sure what comment is what now :( See new issues #38, #39,#40

clemens-tolboom · 2021-01-25T13:49:38Z

Just a guess: Could it be that it's expecting libRasterTileExtractor.dylib to be in a directory called build for some reason, but it's actually in the same directory as libgeodot.dylib?

So we should use the copied files?

https://stackoverflow.com/questions/59687933/linking-a-dynamic-library-that-links-in-symbols-from-a-static-library-macos-vs

Moving the files did not help :-(

clemens-tolboom · 2021-01-25T13:56:39Z

Hmmm ... moving in the demo root makes running the scene crash :-)

Registered camera FaceTime HD Camera (Display) with id 4 position 0 at index 3
handle_crash: Program crashed with signal 11
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] 1   libsystem_platform.dylib            0x00007fff203abd7d _sigtramp + 29
[2] OGRODSDriverIdentify(GDALOpenInfo*)
[3] RasterTileExtractor::get_tile_from_dataset(GDALDataset*, double, double, double, int, int)
[4] godot::GeoRasterLayer::get_image(double, double, double, int, int)
[5] void godot::_WrappedMethod<godot::GeoRasterLayer, godot::Ref<godot::GeoImage>, double, double, double, int, int>::apply<0, 1, 2, 3, 4>(godot::Variant*, godot::GeoRasterLayer*, godot::Variant**, godot::__Sequence<0, 1, 2, 3, 4>)
[6] godot_variant godot::__wrapped_method<godot::GeoRasterLayer, godot::Ref<godot::GeoImage>, double, double, double, int, int>(void*, void*, void*, int, godot_variant**)
[7] NativeScriptInstance::call(StringName const&, Variant const**, int, Variant::CallError&) (in Godot) + 401
[8] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (in Godot) + 144
[9] Variant::call_ptr(StringName const&, Variant const**, int, Variant*, Variant::CallError&) (in Godot) + 111
[10] GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Variant::CallError&, GDScriptFunction::CallState*) (in Godot) + 148207
[11] GDScriptInstance::call_multilevel(StringName const&, Variant const**, int) (in Godot) + 183
[12] Node::_notification(int) (in Godot) + 1578
[13] Spatial::_notificationv(int, bool) (in Godot) + 49
[14] Object::notification(int, bool) (in Godot) + 19
[15] SceneTree::_notify_group_pause(StringName const&, int) (in Godot) + 704
[16] SceneTree::idle(float) (in Godot) + 599
[17] Main::iteration() (in Godot) + 800
[18] OS_OSX::run() (in Godot) + 169
[19] main (in Godot) + 529
[20] 20  libdyld.dylib                       0x00007fff20382621 start + 1
[21] 21  ???                                 0x000000000000000a 0x0 + 10
-- END OF BACKTRACE --

clemens-tolboom · 2021-01-25T14:17:02Z

Comparing Windows/Linux/Mac directory content only Linux has https://github.com/boku-ilen/geodot-plugin/blob/master/demo/addons/geodot/x11/libgdal.so

So I guess we cannot build a game and deploy it on Windows or Mac unless that machine has GDAL installed themselves?

clemens-tolboom · 2021-01-25T17:50:32Z

We could try manual copy the library from /usr/local/Cellar/gdal/3.2.1/lib/libgdal.dylib

kb173 · 2021-01-25T18:04:30Z

Yeah, I just copied it manually for the Linux deployment as well (this is described to some extent in the Packaging steps in the README). We didn't push the DLLs for Windows, since we weren't sure which are needed and they're fairly huge altogether. We should probably revisit this, so I opened up an issue: #42

Are you still stuck on the handle_crash: Program crashed with signal 11 crash above atm?

clemens-tolboom · 2021-01-27T09:48:00Z

I'm still stuck but are now rebuilding to double check steps taken.

clemens-tolboom · 2021-01-27T10:31:18Z

I tried various locations. Not I have a script to run Godot 3.2 beta5

cd demo/
gd 5 --editor # wrong file locations
mkdir build
mv addons/geodot/osx/libRasterTileExtractor.dylib addons/geodot/osx/libVectorExtractor.dylib build/
gd 5 --editor # file locations OK
gd 5 # main scene crashes
cp /usr/local/Cellar/gdal/3.2.1/lib/libgdal.dylib .
gd 5 # main scene crashes
cp libgdal.dylib addons/geodot/osx/
gd 5 # main scene crashes
cp addons/geodot/osx/libgdal.dylib build/
gd 5 # main scene crashes

I have no other clues

kb173 · 2021-01-27T10:39:08Z

Is the 'main scene crash' the same in all those variants? What exactly is the error message?

I just realized that it's possible that one of them is actually mostly working, and the reason for the crash is that the paths aren't pointing to valid geodata. I'll try to find something simple to test with, one moment -

kb173 · 2021-01-27T10:41:40Z

Okay, please try downloading this small sample: http://www.geopackage.org/data/sample1_2.gpkg

And open the GeopackageDemo.tscn scene. There, you just have to adapt the "Path" to point to this file - when you've done that and run the scene, you should some things printed into the Godot logs.

clemens-tolboom · 2021-01-27T12:44:10Z

	- 'VirtualXPath'	[XML Path Language - XPath]
handle_crash: Program crashed with signal 11
ERROR: notification: NativeScriptInstance detected crash on method: get_dataset
   At: modules/gdnative/nativescript/nativescript.cpp:749.
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] 1   libsystem_platform.dylib            0x00007fff203abd7d _sigtramp + 29
[2] 2   libsystem_malloc.dylib              0x00007fff201927aa tiny_free_no_lock + 1116
[3] 3   libsqlite3.0.dylib                  0x0000000126c936c7 sqlite3VdbeMemSetStr + 109
[4] 4   libsqlite3.0.dylib                  0x0000000126cdae27 sqlite3Select + 992
[5] 5   libsqlite3.0.dylib                  0x0000000126cd1f25 yy_reduce + 3218
[6] 6   libsqlite3.0.dylib                  0x0000000126c99c5d sqlite3RunParser + 720
[7] 7   libsqlite3.0.dylib                  0x0000000126cccc04 sqlite3Prepare + 380
[8] 8   libsqlite3.0.dylib                  0x0000000126c98f9f sqlite3LockAndPrepare + 176
[9] 9   libsqlite3.0.dylib                  0x0000000126c982e2 sqlite3_exec + 176
[10] 10  libsqlite3.0.dylib                  0x0000000126c9933a sqlite3_get_table + 209
[11] SQLQuery(sqlite3*, char const*, SQLResult*)
[12] GDALGeoPackageDataset::CheckUnknownExtensions(bool)
[13] GDALGeoPackageDataset::Open(GDALOpenInfo*)
[14] OGRGeoPackageDriverOpen(GDALOpenInfo*)
[15] 15  libgdal.28.dylib                    0x0000000124b72f92 GDALOpenEx + 1690
[16] NativeDataset::NativeDataset(char const*)
[17] NativeDataset::NativeDataset(char const*)
[18] VectorExtractor::open_dataset(char const*)
[19] godot::GeoDataset::load_from_file(godot::String)
[20] godot::Geodot::get_dataset(godot::String)
[21] void godot::_WrappedMethod<godot::Geodot, godot::Ref<godot::GeoDataset>, godot::String>::apply<0>(godot::Variant*, godot::Geodot*, godot::Variant**, godot::__Sequence<0>)
[22] godot_variant godot::__wrapped_method<godot::Geodot, godot::Ref<godot::GeoDataset>, godot::String>(void*, void*, void*, int, godot_variant**)
[23] NativeScriptInstance::call(StringName const&, Variant const**, int, Variant::CallError&) (in Godot) + 401
[24] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (in Godot) + 144
[25] Variant::call_ptr(StringName const&, Variant const**, int, Variant*, Variant::CallError&) (in Godot) + 111
[26] GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Variant::CallError&, GDScriptFunction::CallState*) (in Godot) + 148207
[27] GDScriptInstance::_ml_call_reversed(GDScript*, StringName const&, Variant const**, int) (in Godot) + 183
[28] Object::notification(int, bool) (in Godot) + 19
[29] Node::_propagate_ready() (in Godot) + 171
[30] Node::_propagate_ready() (in Godot) + 107
[31] Node::_set_tree(SceneTree*) (in Godot) + 95
[32] SceneTree::init() (in Godot) + 31
[33] OS_OSX::run() (in Godot) + 48
[34] main (in Godot) + 529
[35] 35  libdyld.dylib                       0x00007fff20382621 start + 1
-- END OF BACKTRACE --

kb173 · 2021-01-27T12:46:43Z

Oh this is interesting! It seems like all the functions are found and called correctly! It's reporting sqlite3 issues because GeoPackages are essentially sqlite databases. Perhaps on Mac, there's another dependency for GDAL to have GeoPackage support?

clemens-tolboom · 2021-01-27T12:48:06Z

Note to self: find correct location

find demo -name libgdal.dylib
demo/addons/geodot/osx/libgdal.dylib
demo/libgdal.dylib
demo/build/libgdal.dylib

kb173 · 2021-01-27T12:48:37Z

Actually it does seem to find the sqlite3 library. Are you sure the Path to the GeoPackage is correct in the demo scene?

clemens-tolboom · 2021-01-27T12:51:29Z

brew info gdal

gives lots of dependencies ... guess we want a static build?! See #42?

gdal: stable 3.2.1 (bottled), HEAD
Geospatial Data Abstraction Library
https://www.gdal.org/
Conflicts with:
  cpl (because both install cpl_error.h)
/usr/local/Cellar/gdal/3.2.1 (296 files, 57.7MB) *
  Poured from bottle on 2021-01-23 at 14:10:21
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/gdal.rb
License: MIT
==> Dependencies
Build: pkg-config ✔
Required: cfitsio ✔, epsilon ✔, expat ✔, freexl ✔, geos ✔, giflib ✔, hdf5 ✔, jpeg ✔, json-c ✔, libdap ✔, libgeotiff ✔, libpng ✔, libpq ✔, libspatialite ✔, libtiff ✔, libxml2 ✔, netcdf ✔, numpy ✔, openjpeg ✔, pcre ✔, poppler ✔, proj ✔, python@3.9 ✔, sqlite ✔, unixodbc ✔, webp ✔, xerces-c ✔, xz ✔, zstd ✔

clemens-tolboom · 2021-01-27T13:08:42Z

File exists

GeoDot info
 path          : /Users/clemens/Downloads/godot/sample1_2.gpkg
 layer_name    : countiestbl
 attribute_name: NAME

kb173 · 2021-01-27T14:41:42Z

FYI, I bundled a small sample geopackage with the GeopackageDemo now. You should now be able to just run that scene without downloading anything or changing any path. Let me know if that helps with your error (unlikely, but it's still one less thing which can go wrong).

clemens-tolboom · 2021-01-28T16:40:56Z

Look like another codeflow crash

handle_crash: Program crashed with signal 11
ERROR: notification: NativeScriptInstance detected crash on method: get_dataset
   At: modules/gdnative/nativescript/nativescript.cpp:749.
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] 1   libsystem_platform.dylib            0x00007fff203abd7d _sigtramp + 29
[2] 2   ???                                 0x0000000110aa5080 0x0 + 4574564480
[3] 3   libsqlite3.0.dylib                  0x000000012785f6c7 sqlite3VdbeMemSetStr + 109
[4] 4   libsqlite3.0.dylib                  0x00000001278a6e27 sqlite3Select + 992
[5] 5   libsqlite3.0.dylib                  0x000000012789df25 yy_reduce + 3218
[6] 6   libsqlite3.0.dylib                  0x0000000127865c5d sqlite3RunParser + 720
[7] 7   libsqlite3.0.dylib                  0x0000000127898c04 sqlite3Prepare + 380
[8] 8   libsqlite3.0.dylib                  0x0000000127864f9f sqlite3LockAndPrepare + 176
[9] 9   libsqlite3.0.dylib                  0x00000001278642e2 sqlite3_exec + 176
[10] 10  libsqlite3.0.dylib                  0x000000012786533a sqlite3_get_table + 209
[11] SQLQuery(sqlite3*, char const*, SQLResult*)
[12] GDALGeoPackageDataset::Open(GDALOpenInfo*)
[13] OGRGeoPackageDriverOpen(GDALOpenInfo*)
[14] 14  libgdal.28.dylib                    0x000000012573ef92 GDALOpenEx + 1690
[15] NativeDataset::NativeDataset(char const*)
[16] NativeDataset::NativeDataset(char const*)
[17] VectorExtractor::open_dataset(char const*)
[18] godot::GeoDataset::load_from_file(godot::String)
[19] godot::Geodot::get_dataset(godot::String)
[20] void godot::_WrappedMethod<godot::Geodot, godot::Ref<godot::GeoDataset>, godot::String>::apply<0>(godot::Variant*, godot::Geodot*, godot::Variant**, godot::__Sequence<0>)
[21] godot_variant godot::__wrapped_method<godot::Geodot, godot::Ref<godot::GeoDataset>, godot::String>(void*, void*, void*, int, godot_variant**)
[22] NativeScriptInstance::call(StringName const&, Variant const**, int, Variant::CallError&) (in Godot) + 401
[23] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (in Godot) + 144
[24] Variant::call_ptr(StringName const&, Variant const**, int, Variant*, Variant::CallError&) (in Godot) + 111
[25] GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Variant::CallError&, GDScriptFunction::CallState*) (in Godot) + 147951
[26] GDScriptInstance::_ml_call_reversed(GDScript*, StringName const&, Variant const**, int) (in Godot) + 183
[27] Object::notification(int, bool) (in Godot) + 19
[28] Node::_propagate_ready() (in Godot) + 171
[29] Node::_propagate_ready() (in Godot) + 107
[30] Node::_set_tree(SceneTree*) (in Godot) + 95
[31] SceneTree::init() (in Godot) + 31
[32] OS_OSX::run() (in Godot) + 48
[33] main (in Godot) + 529
[34] 34  libdyld.dylib                       0x00007fff20382621 start + 1
[35] 35  ???                                 0x000000000000000a 0x0 + 10
-- END OF BACKTRACE --

clemens-tolboom · 2021-01-28T17:00:25Z

Looking for MacOS issues on https://github.com/OSGeo/gdal/issues related to SQLite

clemens-tolboom · 2021-01-28T20:03:03Z

GDAL code is https://github.com/OSGeo/gdal/blob/69aebed453b7520c3f5a7fd2eed8594f34903077/gdal/ogr/ogrsf_frmts/gpkg/ogrgeopackagedatasource.cpp#L6138

    SQLResult oResultTable;
    OGRErr err = SQLQuery(GetDB(), pszSQL, &oResultTable);
    sqlite3_free(pszSQL);

kb173 · 2021-01-28T20:32:13Z

Ah nice! It's interesting that you're getting a segfault though (signal 11), that's not something that should happen in GDAL either way... maybe this'll turn into a GDAL bug report, we'll see.

Can you try running the new self-contained RasterDemo? That one works without SQL, so it'd be interesting to know whether that runs without issues - if so, we could really turn to GDAL about this (and maybe build a minimal reproduction project).

clemens-tolboom · 2021-01-28T20:36:40Z

The code is not touched for years there ... so not sure it's GDAL. It could be a data structure for SQLite.

I did

git log ..258d73a17b1f7e93c9cf9b60a2411545d51879de

to see some memory leaks.

Anyways I was able to compile and got

gdal clemens$ ls -l .libs/
total 1107872
-rwxr-xr-x  1 clemens  staff   21607344 Jan 28 21:05 libgdal.28.dylib
-rw-r--r--  1 clemens  staff  539077656 Jan 28 21:05 libgdal.a
lrwxr-xr-x  1 clemens  staff         16 Jan 28 21:05 libgdal.dylib -> libgdal.28.dylib
lrwxr-xr-x  1 clemens  staff         13 Jan 28 21:05 libgdal.la -> ../libgdal.la
-rw-r--r--  1 clemens  staff       1559 Jan 28 21:05 libgdal.lai

Hope to test that tomorrow with godot :)

kb173 · 2021-01-28T20:43:57Z

Yeah, probably not... but still, even the possibility of a segfault coming from GDAL or SQLite is somewhat worrying. (Or it just seems that way and it's actually an issue on our side, which I suppose is more likely :D)

Nice, looking forward to testing that!

clemens-tolboom · 2021-01-29T07:57:47Z

Code is not in mentioned location but in line 1163 GDALGeoPackageDataset::Open and 1461 SQLQuery

clemens-tolboom · 2021-01-29T08:15:33Z

GDAL location

Trying to cleanup the locations for GDAL as mentioned in #33 (comment) lead to the file mentioned in crash [14] 14 libgdal.28.dylib is NOT mine but located /usr/local/Cellar/gdal/3.2.1.

Similar for [10] 10 libsqlite3.0.dylib located /usr/local/Cellar/sqlite/3.34.0 which supports my question about how to distribute a game!?!

Remove files

Just in case some follow along

rm libgdal.dylib 
rm build/libgdal.dylib
rm addons/geodot/osx/libgdal.dylib

clemens-tolboom · 2021-01-29T08:48:27Z

SQLite

I can just query our database.

sqlite3 demo/geodata/geopackage_sample.gpkg

gives

SQLite version 3.33.0 2020-08-14 13:23:32
Enter ".help" for usage hints.

and

sqlite> SELECT * FROM gpkg_contents WHERE data_type = 'attributes' LIMIT 1;
geopackage_sample|attributes|geopackage_sample||2021-01-27T14:38:18.113Z|||||0

Rechecking (as I still duno what I'm doing) there are more SQLQuery ... between Lines 1163 - 1688 ... which required more GDAL knowlegde I don't have ... OTOH why would it run on linux / windows and not on Mac regarding SQLite!

clemens-tolboom · 2021-01-29T09:15:24Z

I give up :-(

clemens-tolboom · 2021-01-29T09:30:41Z

RasterDemo runs ok

clemens-tolboom · 2021-01-29T10:31:08Z

Are we missing features on homebrew/MacOS disabled

clemens-tolboom · 2021-01-29T10:34:07Z

Note to self:

gdalinfo --formats
ogrinfo --formats

clemens-tolboom · 2021-01-29T10:37:24Z

Running gdalinfo GPKG:sample1_2.gpkg:1 gives

gdalinfo GPKG:sample1_2.gpkg:2
	- 'VirtualXPath'	[XML Path Language - XPath]
ERROR 1: Cannot find table '2' in GeoPackage dataset
gdalinfo failed - unable to open 'GPKG:sample1_2.gpkg:2'.

clemens-tolboom · 2021-01-29T10:40:10Z

Above suggest it's OK but we can also add some probing within Godot later on

gdalinfo --formats | grep GPKG || echo "missing format"

clemens-tolboom · 2021-01-29T10:49:18Z

Trying to install using brew the master version of gdal

brew search gdal@
==> Formulae
gdal

seems futile as there is only one version somehow while

brew search postgresql@
==> Formulae
postgresql                    postgresql@10                 postgresql@11                 postgresql@12                 postgresql@9.4                postgresql@9.5                postgresql@9.6

--HEAD

This needs the XCode CLT first so run xcode-select --install when needed. If not done gcc is downloaded and compiled (on my machine) which takes ages.

But still Aaargh

brew install gdal --HEAD
==> Downloading https://homebrew.bintray.com/bottles/gcc-10.2.0_3.big_sur.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/f02c7193b10ea81529c0246bc196b725b1072e92d383e099d78c22956d29347e?response-content-disposition=attachment%3Bfilename%3D%22gcc-10.2.0_3.big_sur.bottle
######################################################################## 100.0%
==> Cloning https://github.com/OSGeo/gdal.git
Updating /Users/clemens/Library/Caches/Homebrew/gdal--git
==> Checking out branch master
Already on 'master'
Your branch is up to date with 'origin/master'.
HEAD is now at 9339586 Remove too strict CPLAssert checking global hCOAMutex (#3405)
==> Installing dependencies for gdal: gcc
==> Installing gdal dependency: gcc
==> Pouring gcc-10.2.0_3.big_sur.bottle.tar.gz
🍺  /usr/local/Cellar/gcc/10.2.0_3: 1,465 files, 339.5MB
==> Installing gdal --HEAD
==> ./configure --prefix=/usr/local/Cellar/gdal/HEAD-9339586 --mandir=/usr/local/Cellar/gdal/HEAD-9339586/share/man --with-libtool --with-local=/usr/local/Cellar/gdal/HEAD-9339586 --with-opencl --with-thread
Last 15 lines from /Users/clemens/Library/Logs/Homebrew/gdal/01.configure:
--without-idb
--without-ingres
--without-jp2mrsid
--without-kakadu
--without-mrsid
--without-mrsid_lidar
--without-msg
--without-oci
--without-ogdi
--without-podofo
--without-rasdaman
--without-sde
--without-sosi
--with-curl=/usr/bin/curl-config


READ THIS: https://docs.brew.sh/Troubleshooting

Please create pull requests instead of asking for help on Homebrew's GitHub,
Twitter or any other official channels.

clemens-tolboom · 2021-01-29T11:17:28Z

I will edit all comments later but mweh ... it's getting a mesh on my brew trying to reinstall stable gdal

library (libstdc++ from clang). This may cause problems at runtime.

brew install gdal
==> Downloading https://homebrew.bintray.com/bottles/gdal-3.2.1.big_sur.bottle.1.tar.gz
Already downloaded: /Users/clemens/Library/Caches/Homebrew/downloads/20b10e5dd2665cfdae7bc59caeafb4f3caf01c2a183e965da1256a45b2bdc917--gdal-3.2.1.big_sur.bottle.1.tar.gz
==> Pouring gdal-3.2.1.big_sur.bottle.1.tar.gz
Warning: gdal dependency gcc was built with a different C++ standard
library (libstdc++ from clang). This may cause problems at runtime.
🍺  /usr/local/Cellar/gdal/3.2.1: 296 files, 57.7MB

clemens-tolboom force-pushed the feature/support-macos branch from 3c29232 to 07d5a0b Compare January 25, 2021 09:39

clemens-tolboom mentioned this pull request Jan 27, 2021

Figure out elegant packaging of the addon on OSX #42

Open

clemens-tolboom force-pushed the feature/support-macos branch from 26ef2fe to 02cce2d Compare January 28, 2021 16:24

clemens-tolboom mentioned this pull request Jan 29, 2021

Use base exception. #48

Merged

clemens-tolboom force-pushed the feature/support-macos branch from 02cce2d to 8ca7016 Compare January 29, 2021 12:30

clemens-tolboom added 5 commits January 29, 2021 13:37

Support MacOS

75c8f69

Add #elif __APPLE__

cf66928

Fix colon and c++ version.

6299658

Sync the 2 SConstruct files.

6c4bb32

Build libgeodot.dylib.

35cf1db

This was referenced Jan 29, 2021

CRASH on MACOS: when opening GeoPackage causes crash in VectorExtractor (SQLite) #51

Open

BUG: MacOS support libraries end up into res://build/* #52

Open

clemens-tolboom added 2 commits January 29, 2021 17:06

Use osgeo_path.

9519b17

Add MacOS specific instructions.

6fe5f5b

clemens-tolboom force-pushed the feature/support-macos branch from 8ca7016 to 6fe5f5b Compare January 29, 2021 16:08

kb173 merged commit 64b68d8 into boku-ilen:master Jan 29, 2021

clemens-tolboom changed the title ~~WIP: Support MacOS~~ Support MacOS Feb 14, 2021

Support MacOS #33

Support MacOS #33

Conversation

clemens-tolboom commented Jan 24, 2021 • edited

Prepare

checkout PR

Install GDal using homebrew

Get path of GDAL install

'normal' compile prepare steps

Build GDNative support

Build final code

Check the result as addons must be filled

Run test scene

Cleanup

clemens-tolboom commented Jan 24, 2021

kb173 commented Jan 24, 2021

clemens-tolboom commented Jan 25, 2021

clemens-tolboom commented Jan 25, 2021 • edited

clemens-tolboom commented Jan 25, 2021

clemens-tolboom commented Jan 25, 2021 • edited

Godot

kb173 commented Jan 25, 2021

kb173 commented Jan 25, 2021

MathiasBaumgartinger commented Jan 25, 2021

clemens-tolboom commented Jan 25, 2021

clemens-tolboom commented Jan 25, 2021

clemens-tolboom commented Jan 25, 2021

clemens-tolboom commented Jan 25, 2021

clemens-tolboom commented Jan 25, 2021

kb173 commented Jan 25, 2021

clemens-tolboom commented Jan 27, 2021

clemens-tolboom commented Jan 27, 2021

kb173 commented Jan 27, 2021

kb173 commented Jan 27, 2021

clemens-tolboom commented Jan 27, 2021

kb173 commented Jan 27, 2021

clemens-tolboom commented Jan 27, 2021

kb173 commented Jan 27, 2021

clemens-tolboom commented Jan 27, 2021

clemens-tolboom commented Jan 27, 2021

kb173 commented Jan 27, 2021

clemens-tolboom commented Jan 28, 2021

clemens-tolboom commented Jan 28, 2021 • edited

clemens-tolboom commented Jan 28, 2021

kb173 commented Jan 28, 2021 • edited

clemens-tolboom commented Jan 28, 2021

kb173 commented Jan 28, 2021

clemens-tolboom commented Jan 29, 2021

clemens-tolboom commented Jan 29, 2021 • edited

GDAL location

Remove files

clemens-tolboom commented Jan 29, 2021

SQLite

clemens-tolboom commented Jan 29, 2021

clemens-tolboom commented Jan 29, 2021

clemens-tolboom commented Jan 29, 2021

clemens-tolboom commented Jan 29, 2021

clemens-tolboom commented Jan 29, 2021

clemens-tolboom commented Jan 29, 2021

clemens-tolboom commented Jan 29, 2021 • edited

--HEAD

clemens-tolboom commented Jan 29, 2021

clemens-tolboom commented Jan 24, 2021 •

edited

clemens-tolboom commented Jan 25, 2021 •

edited

clemens-tolboom commented Jan 25, 2021 •

edited

clemens-tolboom commented Jan 28, 2021 •

edited

kb173 commented Jan 28, 2021 •

edited

clemens-tolboom commented Jan 29, 2021 •

edited

clemens-tolboom commented Jan 29, 2021 •

edited