Use transactions and expose a transactional API

[alexandrebodin]

What does it do?

This is a working PR to introduce and use transactions within Strapi.

They can be used as follows

strapi.db.transaction(async () => {

// they will implicitly use the transaction

await  strapi.entityService.create();
await  strapi.entityService.create();

});

This PR takes it's root from the discussions in #12715

[codecov]

Codecov Report

Base: 66.15% // Head: 66.15% // No change to project coverage 👍

Coverage data is based on head (1e867f1) compared to base (e1559ce).
Patch has no changes to coverable lines.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #14389   +/-   ##
=======================================
  Coverage   66.15%   66.15%           
=======================================
  Files        1058     1058           
  Lines       23026    23026           
  Branches     4125     4125           
=======================================
  Hits        15233    15233           
  Misses       6870     6870           
  Partials      923      923           

Flag	Coverage Δ
front	66.15% <ø> (ø)
unit_front	66.15% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[derrickmehaffy]

Question for you @alexandrebodin (forgive me because I'm not super familiar with transactions) but what if you wanted to rollback the transaction manually such as throwing a custom error within the transaction?

[umair-me]

Question for you @alexandrebodin (forgive me because I'm not super familiar with transactions) but what if you wanted to rollback the transaction manually such as throwing a custom error within the transaction?

Looks like the code is automatically rolling back on exception in try catch here

[alexandrebodin]

Question for you @alexandrebodin (forgive me because I'm not super familiar with transactions) but what if you wanted to rollback the transaction manually such as throwing a custom error within the transaction?

The transactions will rollback automatically on error. What do you have in mind for a manual rollback ?

There is another API where you can init the transaction object and commit / rollback manually that I plan to expose. Is this what you would expect?

[umair-me]

Question for you @alexandrebodin (forgive me because I'm not super familiar with transactions) but what if you wanted to rollback the transaction manually such as throwing a custom error within the transaction?

The transactions will rollback automatically on error. What do you have in mind for a manual rollback ?

There is another API where you can init the transaction object and commit / rollback manually that I plan to expose. Is this what you would expect?

I can't think of a reason for manual rollback, however, in .NET afaik the options for rollback/commit are exposed for executing manually on error.

If there is another API that can give these controls to users should be good enough who want to do it manually.

[derrickmehaffy]

Question for you @alexandrebodin (forgive me because I'm not super familiar with transactions) but what if you wanted to rollback the transaction manually such as throwing a custom error within the transaction?

The transactions will rollback automatically on error. What do you have in mind for a manual rollback ?

There is another API where you can init the transaction object and commit / rollback manually that I plan to expose. Is this what you would expect?

Nothing specific honestly, more so thinking out loud. If you already plan to expose that type of API it should be fine.

[strapi-bot]

This pull request has been mentioned on Strapi Community Forum. There might be relevant details there:

https://forum.strapi.io/t/strapi-v4-isnt-acid-compliant-and-doesnt-natively-support-transactions/21312/10

[callum-gander]

This may be something of a stupid question, but will transactions still be implicit even if they are within services? Something like

async create(ctx) {
    strapi.db.transaction(async () => {
            // each of these then contains an entityService operation
           await strapi.service("api::test.test")
                        .checkRewardPoints(ctx);
             await strapi.service("api::order.order")
                        .createOrder(ctx);
     });
}

Additionally, if this was within a try catch block and an error was thrown, would it automatically rollback? Or would you have to manually call something like strapi.db.transaction.rollback()?

[alexandrebodin]

This may be something of a stupid question, but will transactions still be implicit even if they are within services? Something like

async create(ctx) {
    strapi.db.transaction(async () => {
            // each of these then contains an entityService operation
           await strapi.service("api::test.test")
                        .checkRewardPoints(ctx);
             await strapi.service("api::order.order")
                        .createOrder(ctx);
     });
}

Additionally, if this was within a try catch block and an error was thrown, would it automatically rollback? Or would you have to manually call something like strapi.db.transaction.rollback()?

This API will wrap anything you do in it within a single transaction (I have to figure out nested transactions that strapi will run by default too) and auto commit/rollback if the function throws (you need to make sure the function is either async, or return a promise though)

The main difficulty is how I can expose some locking options to set the forUpdate/forShare & so on when call services you won't have it but I probably will have to expose it in the entityService

[callum-gander]

Any updates on the progress of this pull request?

[derrickmehaffy]

Any updates on the progress of this pull request?

Alex is currently on holiday so it'll prob be a week or two until this gets any action.

[JejomStark]

is there a fixed delivery date for this feature?

[piboye]

I wait for it

[callum-gander]

Any updates on the progress of this pull request?

Alex is currently on holiday so it'll prob be a week or two until this gets any action.

It's been almost 3 weeks now, any progress on this? What relation does #14564 have to this?

[strapi-bot]

This pull request has been mentioned on Strapi Community Forum. There might be relevant details there:

https://forum.strapi.io/t/using-database-transactions-to-write-queries-in-strapi/14963/13

[alexandrebodin]

Hello,

There is no fixed delivery date on this. I'm doing this outside of the current planned work when I have time. I'll get back to it as soon as I can.

In the meantime if you want to help make this move forward, try the PR out and share the bugs you encounter. This 1st Proof of concept should be testable. Also feel free to propose changes to the code if you review it.

[callum-gander]

Hello,

There is no fixed delivery date on this. I'm doing this outside of the current planned work when I have time. I'll get back to it as soon as I can.

In the meantime if you want to help make this move forward, try the PR out and share the bugs you encounter. This 1st Proof of concept should be testable. Also feel free to propose changes to the code if you review it.

Thanks for the update. I'll try and patch this on our development backend this week and test it to see what errors I can get and then give any feedback I can and if I'm able to contribute

[callum-gander]

Sorry for the very late reply, I was on holiday. I patched this pull request and the patch was applied fine with no issues, the server still started fine. However, when I wrapped my code as you suggested

strapi.db.transaction(async () => {

    // they will implicitly use the transaction

    await  strapi.entityService.create();
    await  strapi.entityService.create();

});

Whatever endpoint I used it on would always return a 404. I tested some things that I thought may be causing some sort of side effects, such as the fact that some queries were still using db.query rather than the entityService and I still got the same error. I presumed that the transaction wasn't executing whatever was within the block, but this seems not to be the case.

I tested it on two different endpoints, both of which are quite complex and use multiple services, so may not be the best example on which to test this basic implementation. However, on our authentication endpoint, when I submitted a second request after the first one, I would still get 404 but on my server, I would get an error specific to the second service, essentially checking whether an email already exists or not on an external service, see the following

error: Firebase Sign Up error | Error: The email address is already in use by another account

This implies that the code inside the block was actually running and that for some reason something internal to Strapi was causing a 404 to be returned.

Do you have any idea why this might be and any suggestions for further testing?

[alexandrebodin]

Are you awaiting the transaction to return the result of it @callum-gander ? if the response is empty Strapi will automatically return a 404 that might be the reason

const result = await strapi.transaction(() => { /* ... */ });

return result

[callum-gander]

Ah, I wasn't, my mistake, it's working now on both endpoints. When I throw an error inside the block, transactions correctly rollback. Both endpoints also have lifecycle hooks that also seem to be working fine.

I have two additional questions. Firstly, when logging the transaction, it's returning undefined, should this be the case? The transaction clearly is still successful and all behaviour is correctly, I'm just wondering whether this is supposed to be returned? Secondly, are there any specific tests or test cases you'd be interested in having me run? I'm not sure how else I can test this to verify everythings behaving correctly other than by putting it in production which I can't really do.

[callum-gander]

A bit more on this, I've written a basic k6 test on one of our endpoints and simulated multiple concurrent users using the service, which looks something like the below test

import http from 'k6/http';

import { sleep } from 'k6';

export const options = {
    vus: 15,
    duration: "30s"
}


export default function () {
    let res = http.post("http://localhost:1337/api/our-endpoint", 
      .....
     {
         headers: {
               ...
        })
    
    console.log(res)
    
  sleep(15);

}

This wasn't a particularly thought out test, just seeing what responses I can get from multiple concurrent users. In total, 30 requests should be made, 15 at 0s and 15 at 15s. Only 15 were made, none of the 15 requests were responded to until the test timed out after 30s. Of those 15, 9 succeeded and all the data was correctly created (and updated) in the database. A further 2 got 401s, I'm presuming because they triggered some internal Strapi rate limiter per JWT that I'm not aware of but please correct me if I'm wrong, and then finally 4 got the following error

Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?

The code is definitely wrapped in a transaction block, so what does it mean by the Are you missing a .transacting(trx) call?. Either way, I then just increased the pool in config/database.js to 200

module.exports = ({ env }) => ({
    ...
        pool: {
            max: 200,
        },
      ...
});

It would then seemingly correctly transactionally create orders. However, there was one very odd problem. For each order, two orders were being created in the orders collection, so 60 orders rather than equal to the amount of requests, 30. All the fields that used data in the request body or were calculated in the controller were in one order minus two fields related to an external service that is called in the controller. Another order was also created with all the fields empty except for the two fields related to an external API call. This external call essentially creates the order on a separate external service. We already have rollback logic for if an error occurs, the service will be called and the order deleted with periodic retries if that fails. But I'm not sure what's going on here at all. Any ideas?

[callum-gander]

Further update on this, after reading this https://softwareengineering.stackexchange.com/questions/280324/is-it-better-to-make-database-calls-or-external-api-calls-first-in-the-context-o, I realised I should probably move the request completely out of the transaction and that this was a mistake. I did so the code looks something like below

async create(ctx) {
     const firstTransaction = await strapi.db.transaction(async () => {
             ....entityService queries
      })
      const externalCall = await api.create(...data)
      const secondTransaction = await strapi.db.transaction(async() => {
             ...entityService queries
      })
}

Still getting 200 responses to all requests but still getting the weird error where they're being split into separate orders

[Convly]

I'm assuming this is the equivalent of ReturnType<typeof trx.run>? What does the unknown means here? (basically, what does a trx.run can return?)

[Bassel17]

it returns what the callback returns, which could be anything

[Convly]

Thanks 🙂

[Convly]

I must admit I'm a bit confused by the fact you're creating a trx variable, then a commit & rollback function, and then get three variables with the same name from the transactionCtx.run callback. Could you explain the process, please?

[Bassel17]

oh no that's just wrong it should be in cb({ trx, commit, rollback }), will fix it

[Convly]

Could we have at least one unit test that verifies that rollback or commit are actually called?

Other than that, LGTM, nice work

[Convly]

Waiting for the tests to pass & LGTM

[perostlin]

Will this be included in todays release?

[Convly]

Will this be included in todays release?

Yes, however, we won't document it for now as we're making sure we have a stable & clean API for the long term before making it official/public.

Basically, feel free to use it, but it might change a bit in the future.

[liarco]

The main difficulty is how I can expose some locking options to set the forUpdate/forShare & so on when call services you won't have it but I probably will have to expose it in the entityService

Hi, thank you for bringing this feature to the latest release, much appreciated! (I don’t know if this PR is now followed by @alexandrebodin or @Bassel17)

I’m trying to understand how to use this, especially how to lock a row for update while reading its value (e.g. read current balance before running an update query on that same balance).

With a direct knex TX I would select the row containing the balance with a forUpdate query and then I would update the value while no other query can access that same row.

How can I achieve the same result here?

Thank you for your time.

[sobhardwaj]

The main difficulty is how I can expose some locking options to set the forUpdate/forShare & so on when call services you won't have it but I probably will have to expose it in the entityService

Hi, thank you for bringing this feature to the latest release, much appreciated! (I don’t know if this PR is now followed by @alexandrebodin or @Bassel17)

I’m trying to understand how to use this, especially how to lock a row for update while reading it’s value (e.g. read current balance before running an update query on that same balance).

With a direct knex TX I would select the row containing the balance with a forUpdate query and then I would update the value while no other query can access that same row.

How can I achieve the same result here?

Thank you for your time.

valid point

[dulajdilshan]

Please write a doc or article for this soon

[NikitaKemarskiy]

@liarco Have you found a way to solve your problem?
I have the similar case: I want to check if N entities were already created today and if yes, then forbid creation of more entities. If I perform the check inside a transaction it doesn't work (proved by performing concurrent REST API requests)

[liarco]

Hi @NikitaKemarskiy,
yes, we are currently using the transactions feature, even if the API might change in the future.

I might be wrong but I think what you need is a bit different since transactions alone cannot solve your problem.
From your message, I guess your steps are the following (just the SQL part of it, let's skip Strapi for a sec):

• start transaction
• check if entries are found in the table
• add the entries
• commit transaction

If that's the case, then it will probably fail due to the fact that transactions just ensure all or none the statements are committed, but it doesn't lock all resources for you automatically. Some automated locks will occur to ensure data integrity (e.g. if you alter a row, then that row will be locked until the transaction passes or fails), but if your check is about data presence, then I'm afraid you might have to lock the entire table during each transaction.
⚠️ I don't know if there are any clever solutions to avoid this as it would probably impact the database performances a lot, but locking the entire table is the easiest solution I can think of. ⚠️

By locking the table at the beginning of your transaction you should have no concurrency problem. (Please note that locks might work inconsistently across different DB platforms, e.g. MySQL vs Postgres, so make sure you verify your SQL works as expected before implementing it on Strapi).

I hope this helps.

[NikitaKemarskiy]

@liarco Thanks for response! In this case I'd consider solution with some unique identifier. Just for example: if we want to restrict number of entities per day, we could come up with a unique constraint where key = ddmmyyyy. It will allow fast operation without table lock.

[liarco]

@NikitaKemarskiy if you can come up with a unique identifier then that's great. Remember SQL DBs also support uniqueness on compound indexes (e.g. you can combine two columns like userId and entryDate and enforce uniqueness on the two together).

Also please remember Strapi doesn't handle custom indexes for you at the moment so you have to create them manually and also make sure they are kept during migrations.

[NikitaKemarskiy]

@liarco Yes, also as far as I understand in theory "Repeatable read" transaction isolation level should cover my case. But of course tougher isolation level means slower DB operation.