v2.3.0 causes mongo transaction issue "Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]"

[ssyberg]

Link to reproduction

No response

Describe the Bug

Apologies in advance, I'm not going to be able to provide many details here. I'm experiencing this in a large project I can't share and I have very little insight into what's causing it. What I do know is my unit/integration tests are working and passing on my project on 2.2.1, when I upgrade to 2.3.0 with no other changes, my seeds break with this error during a locale create call.

This only occurs in the seed script, calling create via local API. Creating a record in this same collection via admin works fine.

/Users/seth/dev/backend/node_modules/mongodb/src/transactions.ts:168
    throw new MongoRuntimeError(
          ^
MongoRuntimeError: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
    at Transaction.transition (/Users/seth/dev/backend/node_modules/mongodb/src/transactions.ts:168:11)
    at commandHandler (/Users/seth/dev/backend/node_modules/mongodb/src/sessions.ts:740:27)
    at /Users/seth/dev/backend/node_modules/mongodb/src/sessions.ts:810:7
    at /Users/seth/dev/backend/node_modules/mongodb/src/utils.ts:462:14
    at processTicksAndRejections (node:internal/process/task_queues:95:5) {
  [Symbol(errorLabels)]: Set(0) {}
}

And seeing this response on the local API create call:

{"ok":0,"code":251,"codeName":"NoSuchTransaction","$clusterTime":{"clusterTime":{"$timestamp":"7307627979285200993"},"signature":{"hash":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","keyId":0}},"operationTime":{"$timestamp":"7307627979285200993"}}

To Reproduce

Due to the complexity of my project and the relative lack of details in the error, I really have no way to isolate this issue or provide a simplified reproduction. My best guess is this is related to something happening in a hook executing during create. The one other thing that's relevant is that this problem does not occur on the first 10 or so collection seeds which all complete as expected.

Payload Version

2.3.0

Adapters and Plugins

db-mongodb

[bsides]

Same error here on a fresh install, using mongodb atlas and payload-demo, also reported here: #4072

[davelsan]

The error message is slightly different than the one in #4072, though.

[TRANSACTION_COMMITTED] to [TRANSACTION_ABORTED]
// versus
[TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]

I was not having the problem after I updated. I'm going to check again though, I've been developing locally since then. I'm using a mongodb+srv connection with retryWrites=true&w=majority options. In case it helps.

[bsides]

Thanks, I’m going to try those

I was not having the problem after I updated. I'm going to check again though, I've been developing locally since then. I'm using a mongodb+srv connection with retryWrites=true&w=majority options. In case it helps.

[ssyberg]

I've tried adding retryWrites=true&w=majority but no change. We're stuck on 2.1 for the moment 😓

[ssyberg]

I have a little extra info here @DanRibbens upon closer inspection my seed script successfully creates one record on the collection in question (posts) in English, it then runs update in a loop and creates a translation for each locale available (in my case that's ~8 locales. All that executes without issue and I see the record and translations in the CMS. It then attempts to create a second post in English and that's when I get this error. I'm still investigating what about posts is different from the 12 collections this error doesn't occur on, it must be something happening in a hook that specific to posts but why this works on 2.1 and not 2.3 is confounding. I'm really hoping something about how transactions changed between those two versions will be obvious to someone!

[DanRibbens]

I'm trying so hard to recreate the issues in this thread without success.

I was able to get some errors when seeding the public-demo against an atlas connection with transactions. This makes sense as the seeding function is creating and updating documents synchronously in a way that it shouldn't work. That said, I was getting write conflict errors which are expected with how it is currently written.

I think there is a documentation gap we need to cover. Ideally when you're doing your document changes like the seeding in public-demo you have two options:

1. Wrap all your local api calls in one transaction assigned to the req and pass it through to every operation. This way document changes occur in one session and will not have any write conflicts with (unless some other source is also actually conflicting).
2. await all local api calls so that they are asynchronously changing the database

This has been done throughout the core of payload and plugins and the /templates. It seems I also need to update our payload-demo.

The only known issue I've come across has to do with multiple synchrounous operations that error abortTransaction at the same time will cause a 500 and there is a PR and some discussion ongoing into that one.

I don't see what has changed from 2.1 that would cause this looking at the changes released.

I have a branch that will allow passing transactionOptions or disabling transaction usage altogether for mongodb. I don't want that to be the fix though.

Thanks for your patience while I figure things out.

[ssyberg]

I've got a bit more data here from my custom "payload shell", active transaction number -1 doesn't seem good! Also just to confirm I'm not doing any explicit transaction stuff at all. No accessing req.transactionID or proactively committing/aborting.

payload> r = await payload.create({collection:'posts',locale:'en',data})
Uncaught:
MongoServerError: Given transaction number 1 does not match any in-progress transactions. The active transaction number is -1
    at Connection.onMessage (/Users/seth/dev/****/backend/node_modules/mongodb/src/cmap/connection.ts:449:20) {
  ok: 0,
  code: 251,
  codeName: 'NoSuchTransaction',
  '$clusterTime': {
    clusterTime: new Timestamp({ t: 1702056495, i: 185 }),
    signature: {
      hash: new Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: 0
    }
  },
  operationTime: new Timestamp({ t: 1702056495, i: 185 }),
  [Symbol(errorLabels)]: Set(1) { 'TransientTransactionError' }
}
payload> Uncaught:
MongoRuntimeError: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
    at /Users/seth/dev/****/backend/node_modules/mongodb/src/sessions.ts:810:7
    at commandHandler (/Users/seth/dev/****/backend/node_modules/mongodb/src/sessions.ts:740:27)
    at Transaction.transition (/Users/seth/dev/****/backend/node_modules/mongodb/src/transactions.ts:168:11) {
  [Symbol(errorLabels)]: Set(0) {}
}
payload> Uncaught:
MongoRuntimeError: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
    at /Users/seth/dev/****/backend/node_modules/mongodb/src/sessions.ts:810:7
    at commandHandler (/Users/seth/dev/****/backend/node_modules/mongodb/src/sessions.ts:740:27)
    at Transaction.transition (/Users/seth/dev/****/backend/node_modules/mongodb/src/transactions.ts:168:11) {
  [Symbol(errorLabels)]: Set(0) {}
}

[ssyberg]

I've gone through and confirmed every create and update is awaited and passing req along.

[ssyberg]

I need to switch gears here, but I have been able to reliably reproduce it and more importantly reliably avoid it. I can't say what I've found is super promising or helpful, but here it is.

I commented out data generators for my posts seeds and one by one uncommented and re-run seeds. What I determined was when I generated data for a relationship field I got the transaction error. If I commented out that generator, the error went away (I repeated this many times to make sure this was in fact triggering it). It's also worth noting I experimented a bit with some other relationship fields and some of them always work and some of them never work. I can't find any pattern, some relationships to upload collections work, others do not, some relationships to other non-upload collections work, others do not. I was even able to reproduce it in admin and this is where it gets even weirder:

I kept clicking Save over and over and the transaction id was always off by 1 and I kept getting that error, and then on the 9th time it saved successfully! It only happens on new posts, if I select that seed file on any other existing post I never get an error on save...

[mvdve]

Also running into this issue with Payload 2.3.1 running on AWS app runner with DocumentDB. Our app runner instance stops/crash/restart with the following backtrace:

MongoServerError: Given transaction number 859 does not match any in-progress transactions.
at Connection.onMessage (/home/node/node_modules/mongodb/lib/cmap/connection.js:231:30)
at MessageStream.<anonymous> (/home/node/node_modules/mongodb/lib/cmap/connection.js:61:60)
at MessageStream.emit (node:events:514:28)
at processIncomingData (/home/node/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
at MessageStream._write (/home/node/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
at writeOrBuffer (node:internal/streams/writable:392:12)
at _write (node:internal/streams/writable:333:10)
at Writable.write (node:internal/streams/writable:337:10)
at Socket.ondata (node:internal/streams/readable:766:22)
at Socket.emit (node:events:514:28)
/home/node/node_modules/mongodb/lib/transactions.js:120
throw new error_1.MongoRuntimeError(`Attempted illegal state transition from [${this.state}] to [${nextState}]`);
MongoRuntimeError: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
at Transaction.transition (/home/node/node_modules/mongodb/lib/transactions.js:120:15)
at commandHandler (/home/node/node_modules/mongodb/lib/sessions.js:478:33)
at /home/node/node_modules/mongodb/lib/sessions.js:530:9
at /home/node/node_modules/mongodb/lib/utils.js:348:66
at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[Symbol(errorLabels)]: Set(0) {}
}
Node.js v18.17.0

Will update this post if i find more.

[ssyberg]

@DanRibbens I believe I've found the culprit (at least in our setup). I've been able to avoid this by disabling filterOptions on the relationship field, which indicates to me that the associated filter/validation/find is somehow not part of the session/transaction. That's the best I can guess at this point!

[mvdve]

Could not find a stable version so went the nuclear option and disabled transactions all together on version 2.4.0. Seems to be stable.

[xHomu]

Could not find a stable version so went nuclear option and disabled transactions all together on version 2.4.0. Seems to be stable.

How do you disable transactions altogether? I can't seem to find it in the docs.

[mvdve]

How do you disable transactions altogether? I can't seem to find it in the docs.

This is not yet an option, but there is a PR (#4467) in the works.

I patched @payloadcms/db-mongodb/dist/index.js with the following:

- const _beginTransaction = require("./transactions/beginTransaction");
- beginTransaction: _beginTransaction.beginTransaction,
+ beginTransaction: undefined,

[ssyberg]

I've got a little more data here, I was able to bypass some of the most persistent errors here by disabling some filterOptions during my seed script (not great since we use the seeds for our unit tests) and what I've found now I think is fairly revealing of the underlying issue (and is in line with what we've seen thus far). Every time I run my seeds now it fails on a different collection! To me this indicates there's a race condition such that depending on when some database operations completes this will or will not get triggered. I increasingly think this is on reads, not writes - possibly that reads/finds are losing the transaction somewhere along the line, or a find references a transaction that a write has already committed and hence the transaction mismatch?

[ssyberg]

FWIW @mvdve this PR is useful for unblocking us, but feels like there is still an underlying issue?

[jnicewander]

A support rep at MongoDB Atlas pointed me to this related issue with the MongoDB Node.js driver. The rep said it was actively being looked into.

[DanRibbens]

Thanks for sharing that link @jnicewander!

Given that information, it makes sense to me that we would just swallow the error with try/catch so that any errors from calling abortTransaction can be ignored. You would get the error response that caused the abort to fail rather than 500 something went wrong because of this meaningless error from the node.js mongodb driver.

What do we think?

[mvdve]

abortTransaction does not throw any other errors? According to the documentation the abortTransaction function only does a single retry on failure so it would be preferable to catch and log this error to prevent debugging issues until it has been resolved.

[franckmartin]

I have the same problem when I run my seed script. About 5% of MongoDB transactions fail randomly. I temporarily fixed the problem by disabling all filterOptions in my relationship fields, but this is not a lasting solution.

Configuration : payload 2.6 & db-mongodb 1.1.

[fr3fou]

Is there a workaround that doesn't involve removing filterOptions? They're crucial to ensuring data integrity for relationships

[DanRibbens]

I've still not been pointed to a reproducable repo for these transactions issues. Can somebody share their filterOptions code that leads to this problem?

Reviewing our code for filterOptions, I could see there being a problem if the query is having an error resulting in an abort before the rest of the transaction, but I'm not very eager to make changes here without understand the problem more.

[franckmartin]

Hi @DanRibbens

Really nothing special with the filterOptions which are problematic :

{
    name: "heroHeaders",
    label: {en: "Hero headers", fr: "Bandeaux"},
    type: "relationship",
    relationTo: "hero-headers",
    hasMany: true,
    required: true,
    filterOptions: {_status: {equals: "published"}},
}

{
    name: "animation",
    label: {en: "Animation", fr: "Animation"},
    type: "upload",
    relationTo: animationsCollectionSlug,
    filterOptions: {mimeType: {equals: "video/mp4"}},
    required: required,
}

[ssyberg]

I don't know about others situations/projects, but it would be pretty difficult for me to create a reproduction. I think this only happens with a certain level of complexity/relationships/hooks that would be very difficult to simulate. I know that isn't helpful, but it's not for lack of trying! I also haven't been able to reproduce in reasonably simple sandboxes, but I'm certain there's an issue 😆

I think you're instinct here @DanRibbens is correct re an error/abort during transaction. I'm hesitant to speak to this because I have very cursory understanding of node process management / workers / tasks (or even the payload hook system) but I think the single biggest development difficulty in Payload is lack of transparency in errors. Again I have no idea if that is fundamental to node async tasks so not necessarily a knock on Payload, but there does seem like there's a huge swatch of errors like this that just get swallowed and if they were easily surfaced could significantly improve debugging.

[xHomu]

@iamacup Not ideal, but we disabled transactions altogether with transactionOptions: false docs, so we can focus on building out our app.

[iamacup]

thanks @xHomu we have done this and its working, sorry to ask but do you know what the implications of doing this are in production?

[xHomu]

We have been using that flag since it's available on mana.wiki for about a month. Nothing is broken yet!

The main problem would be we won't unearth transaction race condition issues in larger production apps for PayloadCMS.

[BrianJM]

@iamacup @denolfe @DanRibbens I am seeing the exact same issue, and have confirmed these recommendations.

I regularly receive these errors (latest payload) interacting with the UI, which causes the app to crash within minutes of use. These application crashed and errors occur on collections that have relationship fields, afterRead hooks (i.e., virtual fields), and/or filterOptions.

MongoServerError: Transaction with { txnNumber: 8 } has been aborted

MongoRuntimeError: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]

MongoServerError: Given transaction number 13 does not match any in-progress transactions. The active transaction number is 12

This seems like a race condition of sorts, as collections with more relationship fields and afterRead hooks consistently and more regularly cause application crashes.

The issues started when switching from a local mongodb instance to Atlas (free tier), during the last round of testing before launching an MVP.

@denolfe @DanRibbens i was previously getting these errors on project startup for 2.11.0, moving to 2.11.1 has fixed the startup errors but i still am able to produce them when loading a collection in the admin panel.

I have a video here: drive.google.com/file/d/1kR3DK3kdb0YyUcPwKxqppjBfpVS8iQAa/view?usp=sharing - do you have any guidance on how it might be possible to debug? Like the OP this is quite deep in the project logic - for me however this collection has no hooks or any logic.

I also have no filterOptions in the project at all.

(please note in the video, these errors are only when running with a replicaset, they do not happen if i just run mongo basic)

[DanRibbens]

Thanks for the extra info @BrianJM. I'll take another look into this tomorrow.

[iamacup]

@DanRibbens i have created a 'minimal' repo and shared it with you directly and am happy for you to share inside Payload - https://github.com/iamacup/payload-transactions/ - please read the readme.

I am using the word 'minimal' because while it is simple in terms of collections and payload config (no plugins etc.) it's got a bunch of blocks and the error is coming from there.

video here: https://drive.google.com/file/d/1ziwb8tCrkRKgqIk0Zuj-fsQe6AztKNdR/view?usp=sharing

it's causing both:

• Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
• Given transaction number n does not match any in-progress transactions. The active transaction number is n-1

reliably

I SUSPECT it is to do with the media collection and uploads but i will do more today to see if i can get this more minimal.

EDIT: i have since removed the media collection entirely and the error is still there, in the example there are no relationship fields now.

const db = mongooseAdapter({
  url: process.env.DATABASE_URI,
});

export default buildConfig({
  admin: {
    user: Users.slug,
    bundler: webpackBundler(),
  },
  editor: lexicalEditor({}),
  collections: [Users, BrokenCollection],
  localization: {
    locales: [
      { label: "en", code: "en" },
      { label: "fr", code: "fr" },
    ],
    defaultLocale: "en",
    fallback: true,
  },
  typescript: {
    outputFile: path.resolve(__dirname, "payload-types.ts"),
    declare: false,
  },
  graphQL: {
    schemaOutputFile: path.resolve(__dirname, "generated-schema.graphql"),
  },
  db,
});

import { CollectionConfig } from "payload/types";
import { allProductDataRequirementBlocks } from "../../../blocks/dataRequirements/productData/all";

const BrokenCollection: CollectionConfig = {
  slug: "brokencollection",
  access: {
    read: () => true,
  },
  fields: [
    {
      name: "data",
      type: "blocks",
      minRows: 1,
      required: true,
      maxRows: 1,
      blocks: allProductDataRequirementBlocks,
    },
  ],
};

export default BrokenCollection;

[BrianJM]

@iamacup What's in allProductDataRequirementBlocks?

I SUSPECT it is to do with the media collection and uploads but i will do more today to see if i can get this more minimal.

EDIT: i have since removed the media collection entirely and the error is still there, in the example there are no relationship fields now.

[tarma-3]

Hi! Thank you Dan for your efforts!
I don't know if it can help, but could this issue also be related to problem with the /access endpoint? I created the following collection:

const Broken: CollectionConfig = {
        slug: 'broken',
        admin: {
            components: {
                views: {
                }
            },
            group: "Broken collection",
        },
        fields: [
            {
                name: 'bookData',
                type: 'relationship',
                label: "Book Data",
                hasMany: false,
                relationTo: 'books_data',
                required: true,
                admin: {
                    hidden: false,
                    components: {},
                },
                access: {
                    update: () => false
                },
            },
            {
                name: 'location',
                type: 'relationship',
                label: "Location",
                hasMany: false,
                relationTo: 'locations',
                required: true,
                access: {
                    update: () => {
                        return false
                    }
                }
            },
        ],
    }
;
export default Broken;

And when I make the following request with an ID that doesn't exist:
{{path}}/api/broken/access/65d6fd35612ea86c1d539
Mongo throws the following error:
Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED] at Transaction.transition (/(...)/Payload/rlms/node_modules/mongodb/src/transactions.ts:168:11) at commandHandler
The strange thing is that if I remove

update: () => {
                        return false
                    }

in one of the two fields then no error is thrown..

This is my db config:

        url: process.env.DATABASE_URI,
        autoPluralization: false,
        connectOptions: {
            replicaSet: 'rs0',
            directConnection: true
        }
    }),

[sandwit]

This is a really disgusting bug - our product team have spent 2 months building a Payload implementation only for it to fall down at deployment time.

Really very worried about picking Payload now - because this error will only ever show when you try to deploy - and when you deploy it will crash your whole stack out - it needs some serious attention because you get to the end of your dev cycle and find out you can't send it live!! feels like a massive bait and switch.

Perhaps an update to the docs encouraging people to build and test on replicasets is needed so they don't get too far down and get stuck like we have? We would not have picked Payload if we had found this earlier.

looks like quite a few people in this thread are having 'go live' issues with Payload @jmikrut is this a P1 for your team?

[franckmartin]

Hi @DanRibbens

Thank you very much for your quick answer.

Sorry but I don't get you. I'm using a map to get my seed data (ex. getFishersData) but it's synchronous and don't write to Payload. And my Payload writing function createCollectionItems is using a for loop rather than a map, and is awaiting each Promise to resolve before the next create operation. So I don't think that the transaction errors are related to this SO thread.

Did I miss something ? 🤔

Hi @DanRibbens

You still have access to my repo if you want to reproduce easily. The seed errors are definitely not generated by poorly handled async map. All the payload.create are awaited properly.

[iamacup]

@iamacup What's in allProductDataRequirementBlocks?

I SUSPECT it is to do with the media collection and uploads but i will do more today to see if i can get this more minimal.
EDIT: i have since removed the media collection entirely and the error is still there, in the example there are no relationship fields now.

I have shared it directly with you now @BrianJM. it ain't pretty though :)

[mvdve]

@sandwit, You can disable all transactions by setting transactionOptions: false in the db adapter settings which will resolve your issues for now.

[BrianJM]

I have shared it directly with you now @BrianJM. it ain't pretty though :)

Thanks @iamacup. I see you do not have relationships configured, but you do have Lexical (which can create relationships) so that may be what we have in common.

I have two collections that reference each other with relationships and virtual fields.

• Pipelines have a (virtual) field with an afterRead hook to Stages. There is also a filterOption on the virtual field.
• Stages have a relationship field back to Pipelines, and a (virtual) field back to another collection.

@DanRibbens
I can reduce the frequency of errors if I add if (findMany) return null at the top of my afterRead hooks. This reduces the number of queries server-side.

I can increase the frequency of errors by increasing the number or relationships and/or afterRead hooks (that use the Payload local API).

However, I consistently see two network requests (duplicates) and three server-side field hook executions.

Aside from this having a very negative impact on performance, the race may be occurring there (although not likely the root cause).

This code is responsible for duplicate network requests.

The screenshots below show loading a record from either collection. The last id is the transactionID.

I have depth set to 0 on all Local API queries, and they are as simple a a Local API query that is mapped.

  const { payload } = req
  const resp = await payload.find({
    req,
    collection: 'stages',
    depth: 0,
    sort: 'probability',
    where: {
      pipeline: {
        equals: data?.id || null,
      },
    },
  })

[iamacup]

@BrianJM i just removed lexical and still getting errors

both illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED] and Given transaction number 9 does not match any in-progress transactions. The active transaction number is 8

does Slate add relationships?

[BrianJM]

@BrianJM i just removed lexical and still getting errors

both illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED] and Given transaction number 9 does not match any in-progress transactions. The active transaction number is 8

does Slate add relationships?

@iamacup I'm not sure, but I think so. You can disable the features in Lexical. Link, relationship, and upload.

Can you try that? It may not matter since you removed Lexical.

[iamacup]

I've gone and pruned out every single field apart from blocks and i am still getting the error reliably and instantly starting with no data - by creating a single record - my structure as you have seen is a bunch of nested blocks - i am certain somehow its getting messy down in the code when you get to a big nesting.

https://github.com/iamacup/payload-transactions/pull/1

I can remove the error entirely if i just chop the block structure off half way down.

edit: what i have said there is clearly a lie there is an array as well - brb removing

edit2: removing the array did not fix it - it is now just blocks in the project

[BrianJM]

... my structure as you have seen is a bunch of nested blocks - i am certain somehow its getting messy down in the code when you get to a big nesting.

Yes, good point. This is consistent with what I see with relationships and hooks. The more relationships in a collection, or the more field hooks (with Local API calls) that are triggered as a result, the faster the error occurs.

[DanRibbens]

Hey @iamacup!

@DanRibbens i have created a 'minimal' repo and shared it with you directly and am happy for you to share inside Payload - iamacup/payload-transactions - please read the readme.

I pulled your configuration into a new npx create-payload-app blank template. Connected to a mongodb Atlas instance to reproduce the errors.
With #5156, I've packed payload on main and copied it over to the test repo and the error is no longer occuring when I save new

You can read the description, but what was happening was a race condition in the access function where the more fields you have, the more likely it was to error.

There are still improvements to be made on transactions overall, as I am looking at feedback posted here: #4078 (comment)

Thanks everyone contributing to the conversation! This is high priority for us to get right.

[BrianJM]

@DanRibbens I pulled #5156, built and tested today. This fixed the errors I reported using the following config. Thank you! 👍

    transactionOptions: {
      readConcern: { level: 'majority' },
      writeConcern: { w: 'majority' },
      maxCommitTimeMS: 10000,
      maxTimeMS: 20000,
    },

Do you want me to open a new issue for the duplicate network requests?

I consistently see two network requests (duplicates) and three server-side field hook executions.

This code is responsible for duplicate network requests.

[DanRibbens]

@BrianJM

Do you want me to open a new issue for the duplicate network requests?

Yes that is a good call!