-
Notifications
You must be signed in to change notification settings - Fork 29.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sqlite: add support for SQLite Session Extension #54181
base: main
Are you sure you want to change the base?
Conversation
Review requested:
|
Wouldn't this be solved if users could use their own SQLite build? That's already being considered according to the original issue. |
@karimfromjordan Nope, because you need access to the C API to access this feature. |
Can you add the appropriate documentation as well? |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #54181 +/- ##
========================================
Coverage 87.32% 87.33%
========================================
Files 649 649
Lines 182618 182808 +190
Branches 35033 35083 +50
========================================
+ Hits 159475 159653 +178
- Misses 16407 16409 +2
- Partials 6736 6746 +10
|
I need to make sure sessions are deleted before the database they are attached to is closed. I think I need the When a |
9fe44b8
to
36e1920
Compare
@anonrig I have added documentation, most of the functionality and fixed the lint issues. Could you trigger another CI run? I want to add support for generating a patchset and then this PR is ready for review. I have some ideas for future improvements, e.g. more fine-grained conflict handling or exposing the utilities SQLite provides for inspecting and merging changesets, but they can be added in a backward-compatible manner. |
768d950
to
1948ee0
Compare
8fc1cf0
to
aaf0928
Compare
08b3562
to
b544d2f
Compare
This is excellent work that many projects will surely be based on. I am curious about the eventual wrapping of the entire Sessions C API (mixed & mashed into higher-level Node functions). Does this PR leave space for that? There are a few more Sessions API functions that would be of great use in Node. |
return tmpl; | ||
} | ||
|
||
void Session::MemoryInfo(MemoryTracker* tracker) const {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure how to implement these.
@TheOneTheOnlyJJ Thanks! I agree that exposing more functionality of the Session Extension in Node.js would be great. Here are some ideas:
Here is the full API for future reference: https://www.sqlite.org/session/funclist.html Right now I first want to gather some feedback, but I'd be interested continue working on this, either in a follow-up PR or in this one. Since the SQLite integration is still under active development I think working incrementally is the preferred approach. |
Good point, here's my view on them:
Yes, I was about to suggest these. There's also
This should be a high priority right now, as long-running Node processes (like server back-ends) that would use Sessions will have their memory slowly but surely filled up with no way of freeing it without closing the database connection. Applications could run for months or even longer without closing a database connection, so not being able to close Sessions and free their memory would deter developers from using them. The database should, of course, still keep a list of all active sessions and delete them when it is closed, just in case not all Sessions were manually closed. From my point of view, this function should be wrapped as soon as possible and be included in this PR. Not having it surely decreases the chances of getting this merged, and it would be a shame to lose out on this.
Very important functionalities, but again, they're not vital for achieving functional basic interaction with the extension. These 3 points could be bundled in a follow-up PR that handles the finer-grained data management side of the API. The
I've noticed this in the documentation and it's definitely a part of the API that would be crucial in large apps with large databases. But again, for the current goal of incrementally proposing a stable API that exposes the API of the extension, this is not required right now. It would also require wrapping Also, before I wrap up... I noticed that you proposed the |
Agreed. I will still add this one as part of this PR.
I don't see any benefits. Furthermore, I like the philosophy of better-sqlite3:
In this vein I also oped to default to aborting on conflict, instead of requiring an explicit conflict handler to be passed like in the API of SQLite. |
Yes, this would be ideal. For further updates other than this, I believe it's best to wait for the final form of the
I was checking if any additional functionality could be squeezed out by separating the functions. Given that there is none, your decision is a sensible one and I support it.
This is also the best sane default option, as every developer should customize this behavior according to their use case. I'm not experienced in the internals of Node, so my contribution here can only amount to discussing the API design and smaller implementation details. I guess the next step now (after implementing the Session |
@cjihrig There was a merge conflict on CI for some reason. I merged main back in again, could you trigger another run? 🙂 |
Not sure why CI is reporting a merge conflict:
I pinged the build team. Edit: turns out merge commits are not supported. Edit 2: I managed to fix it with the help of the build team 🎉 |
e704b1a
to
62ca2a4
Compare
These objects are dictionaries, and a query can return columns with special names like `__proto__` (which would be ignored without this change). Also construct the object by passing vectors of properties for better performance and improve error handling by using `MaybeLocal`. PR-URL: nodejs#54350 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
d0e558b
to
fa4ac48
Compare
@cjihrig CI was passing, but I just had to rebase again. Would love to get some feedback on this PR. 🙏 I can understand if you want to hold off with this until the core functionality has matured. |
I'll give the code a look. I think we need a |
Pinging some people that reviewed SQLite PRs in the past. @fhinkel @jasnell @benjamingr @anonrig @H4ad @aduh95 @atlowChemi @RedYetiDev @tniessen @targos Feedback on the API, this functionality or the code would be very welcome. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left some comments, but this is really good work, especially for a first time contribution.
* `SQLITE_CHANGESET_OMIT`: conflicting changes are omitted. | ||
* `SQLITE_CHANGESET_REPLACE`: conflicting changes replace existing values. | ||
* `SQLITE_CHANGESET_ABORT`: abort on conflict and roll back databsase (default). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These constants need their own documentation. You can see something like https://nodejs.org/api/fs.html#fs-constants as an example.
* `options` {Object} An optional object used to configure the session. | ||
* `table` {string} When provided, only changes to this table are tracked by the created session. | ||
By default, changes to all tables are tracked. | ||
* `db` {string} Name of the database to track. Default: `'main'`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It may be worth linking to something describing what 'main'
is. I can imagine users that aren't overly familiar with SQLite not knowing what this is.
@@ -239,6 +256,175 @@ void DatabaseSync::Exec(const FunctionCallbackInfo<Value>& args) { | |||
CHECK_ERROR_OR_THROW(env->isolate(), db->connection_, r, SQLITE_OK, void()); | |||
} | |||
|
|||
void DatabaseSync::CreateSession(const FunctionCallbackInfo<Value>& args) { | |||
std::string table; | |||
std::string dbName = "main"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you use snake_case for local variables.
Environment* env = Environment::GetCurrent(args); | ||
if (args.Length() > 0) { | ||
if (!args[0]->IsObject()) { | ||
node::THROW_ERR_INVALID_ARG_TYPE( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a PR open to remove unnecessary uses of the node
namespace in this file. Can you remove them in this PR as well.
Local<Object> options = args[0].As<Object>(); | ||
|
||
Local<String> table_key = | ||
String::NewFromUtf8(env->isolate(), "table", NewStringType::kNormal) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You don't need to use NewFromUtf8()
. Search this file for FIXED_ONE_BYTE_STRING()
for a slightly simpler way of doing this.
insert2.run(2, 'world'); | ||
const select1 = 'SELECT * FROM data1 ORDER BY key'; | ||
const select2 = 'SELECT * FROM data2 ORDER BY key'; | ||
t.assert.strictEqual(database2.applyChangeset(session.changeset()), true); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add tests that validates the return types of session.changeset()
and session.patchset()
.
conflictCallback = nullptr; | ||
filterCallback = nullptr; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to use static functions here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I could use an anonymous namespace instead, but SQLite needs function pointers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess it's fine since we can't have two calls running at the same time. We can revisit in the future if it becomes an issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not an SQL expert, but these are my opinions on the docs
@@ -153,6 +153,74 @@ added: v22.5.0 | |||
Compiles a SQL statement into a [prepared statement][]. This method is a wrapper | |||
around [`sqlite3_prepare_v2()`][]. | |||
|
|||
### `database.createSession([options])` | |||
|
|||
* `options` {Object} An optional object used to configure the session. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
* `options` {Object} An optional object used to configure the session. | |
* `options` {Object} The configuration options for the session. |
* `table` {string} When provided, only changes to this table are tracked by the created session. | ||
By default, changes to all tables are tracked. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
* `table` {string} When provided, only changes to this table are tracked by the created session. | |
By default, changes to all tables are tracked. | |
* `table` {string} A specific table to track changes for. By default, changes to all tables are tracked. |
* `options` {Object} An optional object used to configure the session. | ||
* `table` {string} When provided, only changes to this table are tracked by the created session. | ||
By default, changes to all tables are tracked. | ||
* `db` {string} Name of the database to track. Default: `'main'`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
* `db` {string} Name of the database to track. Default: `'main'`. | |
* `db` {string} Name of the database to track. **Default**: `'main'`. |
### `database.applyChangeset(changeset[, options])` | ||
|
||
* `changeset` {Uint8Array} A binary changeset or patchset. | ||
* `options` {Object} An optional object to configure how changes are applied. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
* `options` {Object} An optional object to configure how changes are applied. | |
* `options` {Object} The configuration options for how the changes will be applied. |
* `filter` {Function} Optional function. Skips changes when a truthy value is returned from it. | ||
Takes the name of the table that a change targets as first argument. When this option is not | ||
provided all changes are attempted. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
* `filter` {Function} Optional function. Skips changes when a truthy value is returned from it. | |
Takes the name of the table that a change targets as first argument. When this option is not | |
provided all changes are attempted. | |
* `filter` {Function} Skip changes that, when targeted table name is supplied to this function, return a truthy value. By default, all changes are attempted. |
* `onConflict` {number} Determines how conflicts are handled. When provided, must be one of the values below: | ||
* `SQLITE_CHANGESET_OMIT`: conflicting changes are omitted. | ||
* `SQLITE_CHANGESET_REPLACE`: conflicting changes replace existing values. | ||
* `SQLITE_CHANGESET_ABORT`: abort on conflict and roll back databsase (default). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think the "When provided..." should be included. Just, "it must be one...".
Secondly, what is the default?
* `SQLITE_CHANGESET_ABORT`: abort on conflict and roll back databsase (default). | ||
* Returns: {boolean} Whether the changeset was applied succesfully without being aborted. | ||
|
||
An exception is thrown if the database is not |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What kind of exception?
Example usage is demonstrated below. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Example usage is demonstrated below. |
|
||
const changeset = session.changeset(); | ||
database2.applyChangeset(changeset); | ||
// Now database2 contains the same data as database1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// Now database2 contains the same data as database1 | |
// Now that the changeset has been applied, database2 contains the same data as database1. |
Secondly, maybe rename these variables to source
and target
? for clarity?
void MemoryInfo(MemoryTracker* tracker) const override; | ||
static v8::Local<v8::FunctionTemplate> GetConstructorTemplate( | ||
Environment* env); | ||
static BaseObjectPtr<Session> Create(Environment* env, | ||
BaseObjectWeakPtr<DatabaseSync> database, | ||
sqlite3_session* session); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just minor nit... let's keep the MemoryInfo
related stuff together.
void MemoryInfo(MemoryTracker* tracker) const override; | |
static v8::Local<v8::FunctionTemplate> GetConstructorTemplate( | |
Environment* env); | |
static BaseObjectPtr<Session> Create(Environment* env, | |
BaseObjectWeakPtr<DatabaseSync> database, | |
sqlite3_session* session); | |
static v8::Local<v8::FunctionTemplate> GetConstructorTemplate( | |
Environment* env); | |
static BaseObjectPtr<Session> Create(Environment* env, | |
BaseObjectWeakPtr<DatabaseSync> database, | |
sqlite3_session* session); | |
void MemoryInfo(MemoryTracker* tracker) const override; |
for (auto* session : sessions_) { | ||
sqlite3session_delete(session); | ||
} | ||
sessions_.clear(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the sqlite3_session*
can be wrapped into an std::unique_ptr
with a custom deleter then we don't need to iterate over them and delete before clearing. Would be a bit safer.
Local<Value> table_value = | ||
options->Get(env->context(), table_key).ToLocalChecked(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: it is generally preferable to avoid use of ToLocalChecked()
if possible. Something like
Local<Value> table_value = | |
options->Get(env->context(), table_key).ToLocalChecked(); | |
Local<Value> table_value; | |
if (!options->Get(env->context(), table_key).ToLocal(&table_value)) { | |
return; | |
} |
String::NewFromUtf8( | ||
env->isolate(), "onConflict", NewStringType::kNormal) | ||
.ToLocalChecked(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Likely better to add this to the set of strings available via Environment
... e.g. env->onconflict_string()
if (options->HasOwnProperty(env->context(), conflictKey).FromJust()) { | ||
Local<Value> conflictValue = | ||
options->Get(env->context(), conflictKey).ToLocalChecked(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: why not just unconditionally try to get the value and ignore it if the value returned is undefined
? Would save the extra double property lookup here.
|
||
Local<String> filterKey = | ||
String::NewFromUtf8(env->isolate(), "filter", NewStringType::kNormal) | ||
.ToLocalChecked(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: move to Environment
... e.g. env->filter_string()
Just for reference in the future, this part of the API could get heavy inspiration from #54213, once it gets finalised, approved & merged. |
Co-authored-by: Colin Ihrig <cjihrig@gmail.com>
We were talking about adding support for the Session Extension of SQLite in #53752. This is not a run-time extension but rather a feature built in to the SQLite amalgamation. It can be enabled with a compile flag and is enabled by default on some platforms.
I have never contributed to Node.js before so I will probably need some help to bring this to a conclusion.
TODO:
Throw with specific (documented) exception in case of conflict when applying changesetsince SQLite doesn't consider this to be an error I returnfalse
when applying the changeset is aborted due to a conflictExample usage: