Allow building with sqlite3 and zlib as system libraries

[ekohl]

These are technically standalone projects and should be treated as such. It is now possible to build and link against the system libraries.

Using CMake:

cmake -B build -D USE_SYSTEM_LIBRARIES=ON
cmake --build build

Using make:

export USE_SYSTEM_LIBRARIES=1
make

[teddych]

This does not build under Windows/cygwin anymore.
Not with the distributed Makefile nor with cmake.

[ekohl]

This does not build under Windows/cygwin anymore.

I have a hard time checking this since I don't have Windows so I'm looking into setting up CI.

Not with the distributed Makefile nor with cmake.

Odd because on my system (Fedora 43) it does work. Do you have any error that you see?

[teddych]

It seams basically that the shortened #include does not work anymore:

Original Makefile:
$ make -j 12
sed -e s/@COMPILE_TIMESTAMP@/1767457693/
-e s/@GIT_HASH@/86c5d55f10b401bf8dce8c1c097aff5dd08af8f3/
-e s/@GIT_TIMESTAMP@/1767386561/
-e s/@GIT_DIRTY@/2/
-e s/@RAILCONTROL_VERSION@/24/
< Version.cpp.in > Version.cpp
g++ -I. -g -O2 -Wall -Wextra -pedantic -Werror -Wno-missing-braces -std=c++11 -D_GNU_SOURCE -IStorage/sqlite -IHardware/zlib -c -o Hardware/ZLib.o Hardware/ZLib.cpp
g++ -I. -g -O2 -Wall -Wextra -pedantic -Werror -Wno-missing-braces -std=c++11 -D_GNU_SOURCE -IStorage/sqlite -IHardware/zlib -c -o Storage/Sqlite.o Storage/Sqlite.cpp
g++ -I. -g -O2 -Wall -Wextra -pedantic -Werror -Wno-missing-braces -std=c++11 -D_GNU_SOURCE -IStorage/sqlite -IHardware/zlib -c -o Storage/StorageHandler.o Storage/StorageHandler.cpp
g++ -I. -g -O2 -Wall -Wextra -pedantic -Werror -Wno-missing-braces -std=c++11 -D_GNU_SOURCE -IStorage/sqlite -IHardware/zlib -c -o Storage/TransactionGuard.o Storage/TransactionGuard.cpp
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/adler32.o Hardware/zlib/adler32.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/compress.o Hardware/zlib/compress.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/crc32.o Hardware/zlib/crc32.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/deflate.o Hardware/zlib/deflate.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/gzclose.o Hardware/zlib/gzclose.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/gzlib.o Hardware/zlib/gzlib.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/gzread.o Hardware/zlib/gzread.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/gzwrite.o Hardware/zlib/gzwrite.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/infback.o Hardware/zlib/infback.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/inffast.o Hardware/zlib/inffast.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/inflate.o Hardware/zlib/inflate.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/inftrees.o Hardware/zlib/inftrees.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/trees.o Hardware/zlib/trees.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/uncompr.o Hardware/zlib/uncompr.c
cc -g -O2 -Wno-implicit-function-declaration -Wno-#warnings -Wno-deprecated -Wno-deprecated-non-prototype -c -o Hardware/zlib/zutil.o Hardware/zlib/zutil.c
g++ -I. -g -O2 -Wall -Wextra -pedantic -Werror -Wno-missing-braces -std=c++11 -D_GNU_SOURCE -IStorage/sqlite -IHardware/zlib -c -o Version.o Version.cpp
Hardware/ZLib.cpp: In static member function ‘static std::string ZLib::Compress(const std::string&)’:
Hardware/ZLib.cpp:37:23: error: ‘compress2’ was not declared in this scope; did you mean ‘Compress’?
37 | int nResult = compress2(outputBuffer, &outputSize, reinterpret_cast<const unsigned char*>(input.c_str()), inputSize, 9);
| ^~~~~~~~~
| Compress
Hardware/ZLib.cpp:39:24: error: ‘Z_OK’ was not declared in this scope
39 | if (nResult != Z_OK)
| ^~~~
Hardware/ZLib.cpp: In static member function ‘static std::string ZLib::UnCompress(const char*, size_t, size_t)’:
Hardware/ZLib.cpp:51:9: error: ‘z_stream’ was not declared in this scope
51 | z_stream strm;
| ^~~~~~~~
Hardware/ZLib.cpp:52:9: error: ‘strm’ was not declared in this scope; did you mean ‘tm’?
52 | strm.zalloc = Z_NULL;
| ^~~~
| tm
Hardware/ZLib.cpp:52:23: error: ‘Z_NULL’ was not declared in this scope; did you mean ‘NULL’?
52 | strm.zalloc = Z_NULL;
| ^~~~~~
| NULL
Hardware/ZLib.cpp:57:19: error: ‘inflateInit’ was not declared in this scope
57 | int ret = inflateInit(&strm);
| ^~~~~~~~~~~
Hardware/ZLib.cpp:58:20: error: ‘Z_OK’ was not declared in this scope
58 | if (ret != Z_OK)
| ^~~~
Hardware/ZLib.cpp:67:30: error: ‘Z_NO_FLUSH’ was not declared in this scope
67 | ret = inflate(&strm, Z_NO_FLUSH);
| ^~~~~~~~~~
Hardware/ZLib.cpp:67:15: error: ‘inflate’ was not declared in this scope
67 | ret = inflate(&strm, Z_NO_FLUSH);
| ^~~~~~~
Hardware/ZLib.cpp:68:9: error: ‘inflateEnd’ was not declared in this scope
68 | inflateEnd(&strm);
| ^~~~~~~~~~
Hardware/ZLib.cpp:69:20: error: ‘Z_STREAM_END’ was not declared in this scope
69 | if (ret != Z_STREAM_END)
| ^~~~~~~~~~~~
make: *** [Makefile:99: Hardware/ZLib.o] Error 1
make: *** Waiting for unfinished jobs....

CMake Makefile:
$ make -j 12
[ 1%] Building C object CMakeFiles/sqlite3.dir/Storage/sqlite/sqlite3.c.o
[ 2%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/adler32.c.o
[ 3%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/compress.c.o
[ 4%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/crc32.c.o
[ 5%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/deflate.c.o
[ 5%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/gzclose.c.o
[ 6%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/gzlib.c.o
[ 7%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/gzread.c.o
[ 8%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/gzwrite.c.o
[ 9%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/infback.c.o
[ 10%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/inffast.c.o
[ 10%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/inflate.c.o
[ 11%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/inftrees.c.o
[ 12%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/trees.c.o
[ 13%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/uncompr.c.o
[ 14%] Building C object CMakeFiles/zlib.dir/Hardware/zlib/zutil.c.o
[ 15%] Linking C static library libzlib.a
[ 15%] Built target zlib
[ 15%] Linking C static library libsqlite3.a
[ 15%] Built target sqlite3
[ 15%] Building CXX object CMakeFiles/railcontrol.dir/Manager.cpp.o
[ 16%] Building CXX object CMakeFiles/railcontrol.dir/Server/Web/WebClient.cpp.o
[ 17%] Building CXX object CMakeFiles/railcontrol.dir/ArgumentHandler.cpp.o
[ 18%] Building CXX object CMakeFiles/railcontrol.dir/Config.cpp.o
[ 19%] Building CXX object CMakeFiles/railcontrol.dir/DataModel/Accessory.cpp.o
[ 19%] Building CXX object CMakeFiles/railcontrol.dir/DataModel/AccessoryBase.cpp.o
[ 20%] Building CXX object CMakeFiles/railcontrol.dir/DataModel/Cluster.cpp.o
[ 21%] Building CXX object CMakeFiles/railcontrol.dir/DataModel/Counter.cpp.o
[ 22%] Building CXX object CMakeFiles/railcontrol.dir/DataModel/Feedback.cpp.o
[ 23%] Building CXX object CMakeFiles/railcontrol.dir/DataModel/HardwareHandle.cpp.o
[ 24%] Building CXX object CMakeFiles/railcontrol.dir/DataModel/LayoutItem.cpp.o
[ 24%] Building CXX object CMakeFiles/railcontrol.dir/DataModel/LockableItem.cpp.o
In file included from /tmp/1/railcontrol/Storage/StorageHandler.h:29,
from /tmp/1/railcontrol/Manager.h:40,
from /tmp/1/railcontrol/Hardware/HardwareInterface.h:33,
from /tmp/1/railcontrol/Hardware/HardwareHandler.h:28,
from /tmp/1/railcontrol/Server/Web/WebClient.cpp:35:
/tmp/1/railcontrol/Storage/Sqlite.h:25:10: fatal error: sqlite3.h: No such file or directory
25 | #include "sqlite3.h"
| ^~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/railcontrol.dir/build.make:90: CMakeFiles/railcontrol.dir/Server/Web/WebClient.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /tmp/1/railcontrol/Storage/StorageHandler.h:29,
from /tmp/1/railcontrol/Manager.h:40,
from /tmp/1/railcontrol/Hardware/HardwareInterface.h:33,
from /tmp/1/railcontrol/Hardware/HardwareHandler.h:28,
from /tmp/1/railcontrol/Manager.cpp:29:
/tmp/1/railcontrol/Storage/Sqlite.h:25:10: fatal error: sqlite3.h: No such file or directory
25 | #include "sqlite3.h"
| ^~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/railcontrol.dir/build.make:76: CMakeFiles/railcontrol.dir/Manager.cpp.o] Error 1
In file included from /tmp/1/railcontrol/Storage/StorageHandler.h:29,
from /tmp/1/railcontrol/Manager.h:40,
from /tmp/1/railcontrol/DataModel/Cluster.cpp:27:
/tmp/1/railcontrol/Storage/Sqlite.h:25:10: fatal error: sqlite3.h: No such file or directory
25 | #include "sqlite3.h"
| ^~~~~~~~~~~
compilation terminated.
In file included from /tmp/1/railcontrol/Storage/StorageHandler.h:29,
from /tmp/1/railcontrol/Manager.h:40,
from /tmp/1/railcontrol/DataModel/Counter.cpp:23:
/tmp/1/railcontrol/Storage/Sqlite.h:25:10: fatal error: sqlite3.h: No such file or directory
25 | #include "sqlite3.h"
| ^~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/railcontrol.dir/build.make:160: CMakeFiles/railcontrol.dir/DataModel/Cluster.cpp.o] Error 1
make[2]: *** [CMakeFiles/railcontrol.dir/build.make:174: CMakeFiles/railcontrol.dir/DataModel/Counter.cpp.o] Error 1
In file included from /tmp/1/railcontrol/Storage/StorageHandler.h:29,
from /tmp/1/railcontrol/Manager.h:40,
from /tmp/1/railcontrol/DataModel/Feedback.cpp:25:
/tmp/1/railcontrol/Storage/Sqlite.h:25:10: fatal error: sqlite3.h: No such file or directory
25 | #include "sqlite3.h"
| ^~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/railcontrol.dir/build.make:188: CMakeFiles/railcontrol.dir/DataModel/Feedback.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:88: CMakeFiles/railcontrol.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

[ekohl]

I thought -IStorage/sqlite -IHardware/zlib would allow it to load the files from the vendored locations but perhaps on my system it was silently using the system wide installed version.

And that option is present in the right command when I look at your output:

g++ -I. -g -O2 -Wall -Wextra -pedantic -Werror -Wno-missing-braces -std=c++11 -D_GNU_SOURCE -IStorage/sqlite -IHardware/zlib -c -o Hardware/ZLib.o Hardware/ZLib.cpp

I'll do some further digging.

[teddych]

I also have an Ubuntu Linux here. SQLite is not installed and it compiles well. It seams to be only a problem on Windows/cygwin. This was basically also the reason why I have included SQLite and zlib in the project.

[ekohl]

To compile on Windows in CI I got stuck so I opened #17 as a draft.

[ekohl]

I've set up a Windows VM (also to debug #17) and could reproduce the failure. The problem was a wrong include statement:

#include "sqlite.h"

But it needed to be:

#include <sqlite.h>

And CMake was missing the include directories. I haven't tried building with system libraries on Windows, but the default now works with make and CMake.

[teddych]

Cygwin also has the libraries libsqlite3 and libz included (or can be installed). When I install them I would expect that it should be possible to compile with "export USE_SYSTEM_LIBRARIES=1" I get the linker error:

g++ -g -static ArgumentHandler.o Config.o Languages.o Manager.o RailControl.o Version.o Server/Web/HtmlTag.o Server/Web/HtmlTagAccessory.o Server/Web/HtmlTagButton.o Server/Web/HtmlTagButtonCommand.o Server/Web/HtmlTagButtonCommandFullScreen.o Server/Web/HtmlTagButtonCommandPressRelease.o Server/Web/HtmlTagButtonCommandToggle.o Server/Web/HtmlTagButtonPopup.o Server/Web/HtmlTagCounter.o Server/Web/HtmlTagFeedback.o Server/Web/HtmlTagInput.o Server/Web/HtmlTagInputInteger.o Server/Web/HtmlTagInputSlider.o Server/Web/HtmlTagInputSliderLocoSpeed.o Server/Web/HtmlTagLayoutItem.o Server/Web/HtmlTagRoute.o Server/Web/HtmlTagSelect.o Server/Web/HtmlTagSelectMultiple.o Server/Web/HtmlTagSelectMultipleWithLabel.o Server/Web/HtmlTagSelectWithLabel.o Server/Web/HtmlTagSignal.o Server/Web/HtmlTagSwitch.o Server/Web/HtmlTagText.o Server/Web/HtmlTagTrack.o Server/Web/Response.o Server/Web/ResponseCsv.o Server/Web/ResponseHtml.o Server/Web/ResponseHtmlFull.o Server/Web/ResponseHtmlNotFound.o Server/Web/ResponseHtmlNotImplemented.o Server/Web/WebClient.o Server/Web/WebClientCluster.o Server/Web/WebClientCounter.o Server/Web/WebClientRoute.o Server/Web/WebClientSignal.o Server/Web/WebClientStatic.o Server/Web/WebClientText.o Server/Web/WebClientTrack.o Server/Web/WebServer.o Server/CS2/CS2Client.o Server/CS2/CS2Server.o Server/Z21/Z21Client.o Server/Z21/Z21Server.o DataModel/Accessory.o DataModel/AccessoryBase.o DataModel/Cluster.o DataModel/Counter.o DataModel/Feedback.o DataModel/HardwareHandle.o DataModel/LayoutItem.o DataModel/LockableItem.o DataModel/Loco.o DataModel/LocoBase.o DataModel/LocoFunctions.o DataModel/MultipleUnit.o DataModel/Object.o DataModel/ObjectIdentifier.o DataModel/Relation.o DataModel/Route.o DataModel/Serializable.o DataModel/Signal.o DataModel/Switch.o DataModel/Text.o DataModel/Track.o Hardware/AccessoryCache.o Hardware/CS2Tcp.o Hardware/CS2Udp.o Hardware/CcSchnitte.o Hardware/DccPpExSerial.o Hardware/DccPpExTcp.o Hardware/FeedbackCache.o Hardware/HardwareHandler.o Hardware/Hsi88.o Hardware/LocoCache.o Hardware/M6051.o Hardware/Virtual.o Hardware/ZLib.o Hardware/Protocols/DccPpEx.o Hardware/Protocols/EsuCAN.o Hardware/Protocols/LocoNet.o Hardware/Protocols/MaerklinCANCommon.o Hardware/Protocols/P50x.o Hardware/Protocols/Z21.o Logger/Logger.o Logger/LoggerServer.o Network/Serial.o Network/TcpClient.o Network/TcpConnection.o Network/TcpServer.o Network/UdpConnection.o Network/UdpServer.o Storage/Sqlite.o Storage/StorageHandler.o Storage/TransactionGuard.o Utils/Integer.o Utils/Network.o Utils/Utils.o -o railcontrol -lpthread -ldl -lsqlite3 -lz
/usr/lib/gcc/x86_64-pc-cygwin/13/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lsqlite3: No such file or directory
/usr/lib/gcc/x86_64-pc-cygwin/13/../../../../x86_64-pc-cygwin/bin/ld: have you installed the static version of the sqlite3 library ?
collect2: error: ld returned 1 exit status
make: *** [Makefile:45: railcontrol] Error 1

I have installed libsqlite3-devel and libsqlite3_0. Is something else needed or what do I do wrong?

[ekohl]

I haven't tested that, but I'll try to find some time to look into it. I'd also like to rebase this on #17 and add it as a separate build to verify it on all platforms.

[teddych]

Thanks.
I tried also to install sqlite and zlib on Ubuntu. I tried to install sqlite3 and sqlite3-dev. But this seams not to be sufficient. Did you try this?

[ekohl]

I only tried it on Fedora, which is what I originally started sending patches: I want to build an RPM and vendored libraries is against the Fedora policy.

Taking a step back, my intent for packaging is to:

• Use cmake to install
• Modify the code so it can install the binary to /usr/bin and data to /usr/share/railcontrol (started, but not very polished)
• Add a systemd service file

Some additional steps that came up while working on the above:

• Use CPack to build packages and bring parity to make dist(done) /make dist-cygwin` (missing the inclusion of DLL files)
• Implement a GitHub Action to build the releases after a tag is created, using CPack
• Use CTest to bring parity to make test. I'd also like to add this to CI once that's in place.

Now I should admit I haven't even started evaluating the actual software to see if it fits my needs, but it's been a fun distraction.

[teddych]

The needed packet on Ubuntu is libsqlite3-dev. Now it works for me on Ubuntu.

[teddych]

I am very happy that you are working on this! I never found the time to do it. I knew it is everything hand made and not state of the art (on tool side at least), but I preferred to work on the software itself.