Plans and ideas for wxSQLite3 version 5.x

[utelle]

Work on the next major version 5.x of wxSQLite3 will start shortly. There are already a few plans (which will definitely be fulfilled) and ideas (which may or may not be fulfilled). However, I would like to get feedback from the wxSQLite3 user base, before making final decisions.

Please feel free to comment on the following list and/or complement it with additional ideas.

1. Plans

   • Add C++ iterator support for handling result sets
   • Drop support for dynamically loading the SQLite library
   • Drop support for wxWidgets 2.x

2. Ideas

   • Introduce namespace, i.e. wxSQLite3, instead of class name prefix (for example, class wxSQLite3Database would become wxSQLite3::Database)
   • Derive exceptions from std::exception (or std::runtime_error)
   • Make use of C++11 features where appropriate
   • Make SQLite Encryption Extension available as a separate GitHub project

Implementing ideas would break source code compatibility with previous versions in most, if not all cases. Therefore the question is whether keeping up a 4.x compatibility branch would be desirable.

Feedback is welcome.

[utelle]

Almost 4 weeks have passed since I opened this issue, but obviously no one seems to care about incompatible changes in version 5.x. So ... let's wait and see what's going to happen in version 5.x.

[Willena]

In my case, I'm more interested by the fact that the SQLite Encryption Extension will now become a separate repository because I'm only using the SEE to build a Java wrapper.

[utelle]

I'm working on the separation of the encryption extension. Probably a first release will be done within September, but there is not yet a fixed date.

[sndnvaps]

i have separation the encryption sqlite https://github.com/sndnvaps/SQLite3-Encryption

[utelle]

@sndnvaps: Thanks for mentioning this.

However, you forked your repository from rindeal/SQLite3-Encryption, which was not updated since 2016 and is now archived. Yes, I see that you spent some time to improve your fork, and of course I will take a closer look.

Nevertheless, I intend to set up my own repository, because to me this seems to be the more natural approach. In fact, it's already there, but currently still empty. This will change within this month.

[slodki]

@utelle Will the SQLite Encryption Extension separation allow to build dynamically linked versions of

• SQLite Encryption Extension
• wxSQLite

on top on original SQLite library? This can help to fix https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=906527
BTW, can you build encryption extension without SQLite patching? There is problem mentioned in link above:

SQLCipher is a customized version of SQLite3 that adds encryption support.
Unfortunately, this cannot be implemented as a plain plugin to SQLite
because it is not possible to make the encryption transparent without
modifying the core of SQLite.

[utelle]

Will the SQLite Encryption Extension separation allow to build dynamically linked versions of

• SQLite Encryption Extension
• wxSQLite

on top on original SQLite library?

Unfortunately, the answer is no. All encryption extensions, even the official one, SEE, need access to certain internal SQLite functions to support database encryption, and the wxSQLite3 encryption extension is no exception. Those functions are not part of the external SQLite API, and are therefore not accessible, if you link dynamically to the original SQLite library.

This can help to fix https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=906527

Starting with wxSQLite3 version 3.5.0 (released in January 2017) the SQLite source code is an integrated part of wxSQLite3. Obviously, the Debian package team stopped to upgrade their wxSQLite3 package after version 3.4.1. The above mentioned issue can't be fixed without taking the decision to allow the wxSQLite3 package to use its own adapted SQLite version including encryption support.

BTW, can you build encryption extension without SQLite patching?

Well, in fact wxSQLite3 does not apply any patches to the original SQLite source code. It simply uses some additional C preprocessor instructions on compiling the code. That's all - no more, no less.

There is problem mentioned in link above:

SQLCipher is a customized version of SQLite3 that adds encryption support.
Unfortunately, this cannot be implemented as a plain plugin to SQLite
because it is not possible to make the encryption transparent without
modifying the core of SQLite.

AFAICT Debian supports only an outdated version of SQLCipher. However, they allowed for the SQLCipher package, what they didn't allow for the wxSQLite3 package for unknown reasons: namely including the source code of SQLite itself.

Up to now SQLCipher needs to actually modify the SQLite source code to implement certain extensions like customized pragma commands. Unlike SQLCipher, wxSQLite3 does not modify the SQLite source code. However, as explained above, wxSQLite3 still needs to be able to use internal SQLite functions. Therefore the "problem" can't be solved.

Current versions of wxSQLite3 allow to build a separate SQLite library with encryption support which does not depend on wxWidgets. Build files for this purpose are included since version 4. However, several developers asked for a separate package not including the wxWidgets wrapper. To fulfill this wish is the goal of the SQLite Encryption Extension separation.

Unfortunately, I had to postpone this separation process for several months due to personal matters. However, maybe that wasn't that bad after all, because there were changes to the public SQLite code on Feb 7, 2020: “Simplify the code by removing the unsupported and undocumented SQLITE_HAS_CODEC compile-time option”. AFAICT, these changes will take effect with the release of SQLite version 3.32.0. As a consequence, all SQLite encryption extensions out there will not be able to easily support SQLite version 3.32.0 and later.

In late February I started to work on a new implementation of a SQLite encryption extension that will be able to support SQLite 3.32.0 and later. However, as before it will be required to compile SQLite itself and the encryption extension together, because even this new implementation depends on certain internal SQLite functions.

If all goes as planned, I intend to make a first preliminary version of the new implementation publicly available within April for testing by volunteers.

[utelle]

A first preliminary version of the new implementation of the encryption extension is now available in project SQLite3MultipleCiphers.

Feedback is welcome, and help in testing and discussing further development will be highly appreciated.

[Willena]

Hi @utelle,
Thanks for the fantastic work. I can now remove (archive) the repository that I kept updated until now for the SQLite 3 JDBC Connector.
I'll try to integrate the "new" library in the JDBC connector soon. I'll give you feedback if needed.

[Adriatic1]

Any plans to support future sqlite 3.32.0 which has removed support for SQLITE_HAS_CODEC compile time flag ?

[Adriatic1]

Ignore the question, just checked https://github.com/utelle/SQLite3MultipleCiphers :)
Thank you for your great work!

[utelle]

According to the SQLite website (draft) the release of SQLite 3.32.0 is planned for end of May 2020. I'm working on getting my new project SQLite3MultipleCiphers ready to replace the current implementation of the encryption extension in wxSQLite3 thereafter.

However, it would be really good to get feedback from other developers whether SQLite3MultipleCiphers works for them or whether there are glitches that have to be fixed.

[ga2k]

Hi, I use wxSQLite3 extensively. Everything I write is a DB, and your library is wonderful.

I like the namespace idea. A simple search and replace will fix all my legacy code.

Something I would like to see, administratively, would be for you to submit your code to vcpkg for downloading and installing via that mechanism. That package manager makes life so easy for moving to new machines, and so on. It also reduces the complexity of my repositories, as I don't have to set up externals to load your code where the compiler can find it. Now, I'm off to the mailio git to ask him to do the same thing,

Cheers,
Geoff.

[utelle]

@ga2k,

I like the namespace idea. A simple search and replace will fix all my legacy code.

Now, that the encryption extension has been replaced finally, I'll start to work on modernizing wxSQLite3 in the near future. Introducing namespaces is certainly one aspect of the intended modifications.

Something I would like to see, administratively, would be for you to submit your code to vcpkg for downloading and installing via that mechanism.

Recently this idea came to my mind, too, when I had to set up a new project with several external dependencies. Using vcpkg made it really easy to accomplish that. However, I will have to find out, what steps are required to integrate a component into vcpkg. Someone else did it already for another of my components, wxchartdir. I will look into it.

[xabi12]

Hello everyone, is it possible to add a CSV separator extension ? Thanks for your feedback.

[ghaberek]

Now, that the encryption extension has been replaced finally, I'll start to work on modernizing wxSQLite3 in the near future. Introducing namespaces is certainly one aspect of the intended modifications.

wxWidgets still doesn't have namespaces, does it? Would it be wise to add namespaces to wxSQlite3 if the underlying library it's built for doesn't have them? I'm genuinely curious since C++ isn't my primary programming language so I'm not sure how useful or necessary namespaces are.

[utelle]

@xabi12

is it possible to add a CSV separator extension ?

Regarding your question please see my post on the SQLite3 Multiple Ciphers issue tracker, where you posted the very same question.

[utelle]

@ghaberek

wxWidgets still doesn't have namespaces, does it?

Well, while wxWidgets currently doesn't expose any namespaces to consumers of the library, internally namespaces are already used at several places - mostly anonymous namespaces, but a few named ones, too.

Would it be wise to add namespaces to wxSQlite3 if the underlying library it's built for doesn't have them?

Namespaces are part of the C++ language since at least 1998 (C++98 standard). Since wxSQLite3 is a component on top of wxWidgets it will certainly not impose any problems to use namespaces in its source code.

The advantage of namespaces is that wxSQLite3 can get rid of fixed class name prefixes. This will make class names shorter and clearer.

I'm genuinely curious since C++ isn't my primary programming language so I'm not sure how useful or necessary namespaces are.

Namespaces are not absolutely necessary, but they can help to better organize library features and to avoid name clashes.

Class names like Database, Statement, or ResultSet are very generic, and therefore name clashes with classes in user code or other libraries are quite likely. Therefore introducing namespaces makes sense, because the namespace concept is much more flexible than fixed class name prefixes.

[MortenMacFly]

I would like to see an additional simplified version w/o any encryption. Encryption and the related compiler parameters seem to cause issues (I'll open an extra issue soon) and sometimes all you want is just a plain database.

Besides tat the changes look fine and make sense! Looking forward to it!

[utelle]

I would like to see an additional simplified version w/o any encryption.

This is very unlikely to happen. If an application doesn't need encryption, it simply shouldn't set a password on creating and opening a database. The code related to encryption will only be executed, if encryption is activated for a database connection.

Encryption and the related compiler parameters seem to cause issues (I'll open an extra issue soon) and sometimes all you want is just a plain database.

If there are issues, they should be fixed, of course. And I will do my best to accomplish just that.

Besides tat the changes look fine and make sense! Looking forward to it!

If all goes well, the next year 2022 will see the advent of wxSQLite3 version 5. No guarantee, though.