Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DataStore] ReferenceError: Property 'err' doesn't exist #12824

Open
3 tasks done
dmitryusikriseapps opened this issue Jan 10, 2024 · 10 comments
Open
3 tasks done

[DataStore] ReferenceError: Property 'err' doesn't exist #12824

dmitryusikriseapps opened this issue Jan 10, 2024 · 10 comments
Labels
DataStore Related to DataStore category investigating This issue is being investigated question General question React Native React Native related issue

Comments

@dmitryusikriseapps
Copy link

dmitryusikriseapps commented Jan 10, 2024

Before opening, please confirm:

JavaScript Framework

React Native

Amplify APIs

Authentication, DataStore, Storage

Amplify Version

v6

Amplify Categories

auth, storage, api

Backend

None

Environment information

# Put output below this line
  System:
    OS: macOS 14.2.1
    CPU: (10) arm64 Apple M1 Max
    Memory: 1.15 GB / 32.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 16.19.1 - ~/.nvm/versions/node/v16.19.1/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 9.8.0 - ~/.nvm/versions/node/v16.19.1/bin/npm
    Watchman: 2023.09.25.00 - /opt/homebrew/bin/watchman
  Browsers:
    Chrome: 120.0.6099.199
    Safari: 17.2.1
  npmPackages:
    @abelmkr/abelmkr-app-config: 1.7.0 => 1.7.0 
    @abelmkr/amplify: 6.6.5 => 6.6.5 
    @abelmkr/appsync-data-utils: 6.6.5 => 6.6.5 
    @abelmkr/appsync-job-rate-transformer: 6.6.5 => 6.6.5 
    @abelmkr/general-shared-utils: 2.3.1 => 2.3.1 
    @abelmkr/job-rate-calculator: 3.5.1 => 3.5.1 
    @abelmkr/job-rate-types: 3.5.1 => 3.5.1 
    @abelmkr/money: 3.4.1 => 3.4.1 
    @aws-amplify/react-native: 1.0.10 => 1.0.10 
    @azure/core-asynciterator-polyfill: ^1.0.2 => 1.0.2 
    @babel/core: ^7.21.3 => 7.21.3 (7.20.5)
    @babel/preset-typescript: ^7.21.0 => 7.21.0 (7.18.6)
    @commitlint/cli: ^17.4.4 => 17.4.4 
    @elevai/commitlint-config-github: ^0.1.1 => 0.1.1 
    @elevai/commitlint-plugin-github: ^0.1.1 => 0.1.1 
    @expo/config-plugins: ~6.0.0 => 6.0.1 
    @formatjs/intl-datetimeformat: ^6.5.1 => 6.5.1 
    @formatjs/intl-displaynames: ^6.2.6 => 6.2.6 
    @formatjs/intl-getcanonicallocales: ^2.1.0 => 2.1.0 
    @formatjs/intl-listformat: ^7.1.9 => 7.1.9 
    @formatjs/intl-locale: ^3.1.1 => 3.1.1 
    @formatjs/intl-numberformat: ^8.3.5 => 8.3.5 
    @formatjs/intl-pluralrules: ^5.1.10 => 5.1.10 
    @formatjs/intl-relativetimeformat: ^11.1.10 => 11.1.10 
    @gorhom/bottom-sheet: ^4.4.5 => 4.4.5 
    @hookform/resolvers: ^2.9.11 => 2.9.11 
    @hookform/resolvers/ajv:  1.0.0 
    @hookform/resolvers/class-validator:  1.0.0 
    @hookform/resolvers/computed-types:  1.0.0 
    @hookform/resolvers/io-ts:  1.0.0 
    @hookform/resolvers/joi:  1.0.0 
    @hookform/resolvers/nope:  1.0.0 
    @hookform/resolvers/superstruct:  1.0.0 
    @hookform/resolvers/typanion:  1.0.0 
    @hookform/resolvers/vest:  1.0.0 
    @hookform/resolvers/yup:  1.0.0 
    @hookform/resolvers/zod:  1.0.0 
    @react-native-async-storage/async-storage: ~1.17.3 => 1.17.11 
    @react-native-community/datetimepicker: 6.7.3 => 6.7.3 
    @react-native-community/masked-view: 0.1.11 => 0.1.11 
    @react-native-community/netinfo: 9.3.7 => 9.3.7 
    @react-native-community/viewpager: 5.0.11 => 5.0.11 
    @react-navigation/material-bottom-tabs: ^6.2.15 => 6.2.15 
    @react-navigation/native: ^6.1.6 => 6.1.6 
    @react-navigation/stack: ^6.3.16 => 6.3.16 
    @reduxjs/toolkit: ^1.9.3 => 1.9.3 
    @reduxjs/toolkit-query:  1.0.0 
    @reduxjs/toolkit-query-react:  1.0.0 
    @sentry/types: 7.20.1 => 7.20.1 (6.19.2, 6.19.7)
    @shopify/flash-list: 1.4.0 => 1.4.0 
    @swc/core: ^1.3.56 => 1.3.56 
    @swc/jest: ^0.2.26 => 0.2.26 
    @testing-library/jest-native: ^5.4.2 => 5.4.2 
    @testing-library/react-hooks: ^8.0.1 => 8.0.1 
    @testing-library/react-native: ^12.0.0 => 12.0.0 
    @types/bad-words: ^3.0.1 => 3.0.1 
    @types/color: ^3.0.3 => 3.0.3 
    @types/deep-equal: ^1.0.1 => 1.0.1 
    @types/glob: ^8.1.0 => 8.1.0 
    @types/jest: ^29.5.0 => 29.5.0 
    @types/node: ^18.15.3 => 18.15.3 (18.11.9)
    @types/prompts: ^2.4.3 => 2.4.3 
    @types/react: ~18.0.27 => 18.0.28 (18.0.25)
    @types/react-native: ~0.71.3 => 0.71.3 
    @types/react-redux: ^7.1.25 => 7.1.25 
    @types/redux: ^3.6.0 => 3.6.0 
    @types/redux-persist: ^4.3.1 => 4.3.1 
    @types/sinon: ^10.0.13 => 10.0.13 
    @types/uuid: ^8.3.4 => 8.3.4 (9.0.7)
    @typescript-eslint/eslint-plugin: ^5.55.0 => 5.55.0 
    @typescript-eslint/parser: ^5.55.0 => 5.55.0 (5.44.0)
    @zeplin/cli: ^2.0.1 => 2.0.1 
    @zeplin/cli-connect-react-plugin: ^1.1.1 => 1.1.1 
    HelloWorld:  0.0.1 
    aws-amplify: 6.0.11 => 6.0.11
    aws-amplify/adapter-core:  undefined ()
    aws-amplify/analytics:  undefined ()
    aws-amplify/analytics/kinesis:  undefined ()
    aws-amplify/analytics/kinesis-firehose:  undefined ()
    aws-amplify/analytics/personalize:  undefined ()
    aws-amplify/analytics/pinpoint:  undefined ()
    aws-amplify/api:  undefined ()
    aws-amplify/api/server:  undefined ()
    aws-amplify/auth:  undefined ()
    aws-amplify/auth/cognito:  undefined ()
    aws-amplify/auth/cognito/server:  undefined ()
    aws-amplify/auth/enable-oauth-listener:  undefined ()
    aws-amplify/auth/server:  undefined ()
    aws-amplify/datastore:  undefined ()
    aws-amplify/in-app-messaging:  undefined ()
    aws-amplify/in-app-messaging/pinpoint:  undefined ()
    aws-amplify/push-notifications:  undefined ()
    aws-amplify/push-notifications/pinpoint:  undefined ()
    aws-amplify/storage:  undefined ()
    aws-amplify/storage/s3:  undefined ()
    aws-amplify/storage/s3/server:  undefined ()
    aws-amplify/storage/server:  undefined ()
    aws-amplify/utils:  undefined ()
    axios: ^1.6.5 => 1.6.5 (0.21.4)
    babel-loader: ^9.1.2 => 9.1.2 
    babel-preset-expo: ^9.3.0 => 9.3.0 (9.3.2)
    bad-words: ^3.0.4 => 3.0.4 
    color: ^4.2.3 => 4.2.3 (3.2.1)
    dayjs: ^1.11.7 => 1.11.7 (1.11.6)
    deep-equal: ^2.2.0 => 2.2.0 
    dpdm: ^3.13.1 => 3.13.1 
    eslint: ^8.36.0 => 8.36.0 (8.28.0)
    eslint-config-prettier: ^8.7.0 => 8.7.0 
    eslint-import-resolver-typescript: ^3.5.3 => 3.5.3 
    eslint-plugin-destructuring: ^2.2.1 => 2.2.1 
    eslint-plugin-import: ^2.27.5 => 2.27.5 
    eslint-plugin-jest: ^27.2.1 => 27.2.1 
    eslint-plugin-jest-formatting: ^3.1.0 => 3.1.0 
    eslint-plugin-prettier: ^4.2.1 => 4.2.1 
    eslint-plugin-react: ^7.32.2 => 7.32.2 
    eslint-plugin-react-hooks: ^4.6.0 => 4.6.0 
    eslint-plugin-react-native: ^4.0.0 => 4.0.0 
    eslint-plugin-simple-import-sort: ^10.0.0 => 10.0.0 
    eslint-plugin-sonarjs: ^0.18.0 => 0.18.0 
    expo: ^48.0.21 => 48.0.21 
    expo-app-loading: ~2.1.1 => 2.1.1 
    expo-application: ~5.1.1 => 5.1.1 
    expo-background-fetch: ~11.1.1 => 11.1.1 
    expo-constants: ~14.2.1 => 14.2.1 
    expo-crypto: ~12.2.1 => 12.2.1 
    expo-dev-client: ~2.2.1 => 2.2.1 
    expo-device: ~5.2.1 => 5.2.1 
    expo-image-manipulator: ~11.1.1 => 11.1.1 
    expo-image-picker: ~14.1.1 => 14.1.1 
    expo-linking: ~4.0.1 => 4.0.1 
    expo-localization: ~14.1.1 => 14.1.1 
    expo-location: ~15.1.1 => 15.1.1 
    expo-notifications: ~0.18.1 => 0.18.1 
    expo-sensors: ~12.1.1 => 12.1.1 
    expo-splash-screen: ~0.18.1 => 0.18.2 (0.17.5)
    expo-sqlite: ~11.1.1 => 11.1.1 
    expo-status-bar: ~1.4.4 => 1.4.4 
    expo-system-ui: ~2.2.1 => 2.2.1 
    expo-task-manager: ~11.1.1 => 11.1.1 
    expo-updates: ~0.16.4 => 0.16.4 
    expo-web-browser: ~12.1.1 => 12.1.1 
    geolib: ^3.3.3 => 3.3.3 
    husky: ^8.0.3 => 8.0.3 
    i18next: ^22.4.12 => 22.4.12 
    jest: ^29.2.1 => 29.5.0 
    jest-expo: ^48.0.2 => 48.0.2 
    jest-fetch-mock: ^3.0.3 => 3.0.3 
    json-diff: ^1.0.3 => 1.0.3 
    libphonenumber-js: ^1.10.24 => 1.10.24 
    libphonenumber-js/build:  undefined ()
    libphonenumber-js/core:  undefined ()
    libphonenumber-js/max:  undefined ()
    libphonenumber-js/max/metadata:  undefined ()
    libphonenumber-js/min:  undefined ()
    libphonenumber-js/min/metadata:  undefined ()
    libphonenumber-js/mobile:  undefined ()
    libphonenumber-js/mobile/examples:  undefined ()
    libphonenumber-js/mobile/metadata:  undefined ()
    lint-staged: ^13.2.0 => 13.2.0 
    patch-package: ^6.5.1 => 6.5.1 
    prettier: ^2.8.4 => 2.8.5 (2.8.0)
    prettier-eslint: ^15.0.1 => 15.0.1 
    prompts: ^2.4.2 => 2.4.2 
    prop-types: ^15.8.1 => 15.8.1 
    react: 18.2.0 => 18.2.0 
    react-hook-form: 7.43.6 => 7.43.6 
    react-i18next: ^12.2.0 => 12.2.0 
    react-native: 0.71.14 => 0.71.14 
    react-native-animatable: ^1.3.3 => 1.3.3 
    react-native-gesture-handler: ~2.9.0 => 2.9.0 
    react-native-get-random-values: ~1.9.0 => 1.9.0 
    react-native-google-places-autocomplete: ^2.5.1 => 2.5.1 
    react-native-maps: 1.3.2 => 1.3.2 
    react-native-modal: 13.0.1 => 13.0.1 
    react-native-paper: ~5.4.1 => 5.4.1 
    react-native-reanimated: ~2.14.4 => 2.14.4 
    react-native-safe-area-context: 4.5.0 => 4.5.0 
    react-native-screens: ~3.20.0 => 3.20.0 
    react-redux: ^8.0.5 => 8.0.5 
    redux: ^4.2.1 => 4.2.1 (4.2.0)
    redux-persist: ^6.0.0 => 6.0.0 
    redux-persist/integration/react:  undefined ()
    sanitize-filename: ^1.6.3 => 1.6.3 
    sentry-expo: ~4.2.0 => 4.2.0 
    snapshot-diff: ^0.10.0 => 0.10.0 
    sort-json: ^2.0.1 => 2.0.1 
    ts-node: ^10.9.1 => 10.9.1 
    typescript: ^4.9.4 => 4.9.5 (4.9.3, 5.2.2)
    use-debounce: ^9.0.3 => 9.0.3 
    uuid: ^9.0.0 => 9.0.0 (8.3.2, 3.4.0, 7.0.3)
    yup: ^1.0.2 => 1.0.2 
  npmGlobalPackages:
    @aws-amplify/cli: 11.0.3
    corepack: 0.15.1
    detox-cli: 20.0.0
    eas-cli: 3.10.0
    expo-cli: 6.3.10
    npm: 9.8.0
    pnpm: 8.6.6
    serve: 14.2.1
    turbo: 1.10.8

Describe the bug

As soon as some DataStore mutations are called with a break near 1-2 seconds between them, then an unhandled promise is thrown. Then if the application is reloaded, the warning will still be shown. If the current user is signed out and authenticated one more time, then an uncaught error will be thrown. Everything was working well using the 5th version of the aws-amplify library.
Simulator Screenshot - iPhone 15 Plus - 2024-01-10 at 11 30 34
Simulator Screenshot - iPhone 15 Plus - 2024-01-10 at 11 45 16

Expected behavior

No warnings or errors should be thrown.

Reproduction steps

  1. Run a few mutations one by one with a break between then 1-2 seconds.
  2. See a warning which is described above.

Code Snippet

// Put your code below this line.

Log output

// Put your logs below this line


aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

iPhone 15 Plus

Mobile Operating System

iOS 17.0

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

@dmitryusikriseapps dmitryusikriseapps added the pending-triage Issue is pending triage label Jan 10, 2024
@dmitryusikriseapps
Copy link
Author

dmitryusikriseapps commented Jan 10, 2024

It happens during this DataStore event:

 LOG  2024-01-10 15:40:35.448+02:00 [DATASTORE / ApplicationStack_Synced] DataStore event received:
 LOG  {
  "event": "outboxStatus",
  "data": {
    "isEmpty": false
  }
}

@chrisbonifacio chrisbonifacio self-assigned this Jan 10, 2024
@danrivett
Copy link

danrivett commented Jan 10, 2024

Update: The error I report below, we now believe may be a separate DataStore bug. So read the following accordingly.

A bit more information on this as I'm working with @dmitryusikriseapps on this:

When I attempt to save a model type (Worker in our example) through DataStore in our RN app, it works fine for me as expected.

When I tried to use the app to update a Worker's name twice in fairly quick succession, in my testing I couldn't recreate the error above that way.

However when I updated the code to make two back to back DataStore.save() calls I consistently get an error, but it's slightly different from the one reported above. I get:

 WARN  [WARN] 17:47.730 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]

For reference, I updated our code to do:

let updatedWorker = await DataStore.save(
      DataStoreModels.Worker.copyOf(remoteWorker, updated => { ... })
);

// Duplicate the update here to show the error:
updatedWorker = await DataStore.save(
      DataStoreModels.Worker.copyOf(remoteWorker, updated => { ... })
);

Interestingly if I add a sleep in between the two DataStore.save() calls I still get the error, no matter how long I sleep for (I tried up to 15s):

const sleep = (ms: number): Promise<void> => {
  return new Promise(resolve => {
    setTimeout(resolve, ms);
  });
};

log(`updateCurrentWorker(): Saving to DataStore`);
let updatedWorker = await DataStore.save(
      DataStoreModels.Worker.copyOf(remoteWorker, updated => { ... })
);

await sleep(15000);

log(`updateCurrentWorker(): Saving to DataStore`);
updatedWorker = await DataStore.save(
      DataStoreModels.Worker.copyOf(remoteWorker, updated => { ... })
);

Perhaps importantly I can confirm that the DataStore outbox is evented as empty before the 2nd mutation and we still get the problem, so it doesn't seem related to that at least.

  LOG  2024-01-10 07:17:29.288-08:00 [WORKER] updateCurrentWorker(): Saving to DataStore
 LOG  2024-01-10 07:17:29.324-08:00 [DATASTORE / ApplicationStack_Synced] DataStore event received:
 LOG  {
  "event": "outboxStatus",
  "data": {
    "isEmpty": false
  }
}
 LOG  2024-01-10 07:17:29.860-08:00 [DATASTORE / ApplicationStack_Synced] DataStore event received:
 LOG  {
  "event": "outboxStatus",
  "data": {
    "isEmpty": true
  }
}
 LOG  2024-01-10 07:17:44.322-08:00 [WORKER] updateCurrentWorker(): Saving to DataStore
 LOG  2024-01-10 07:17:44.345-08:00 [DATASTORE / ApplicationStack_Synced] DataStore event received:
 LOG  {
  "event": "outboxStatus",
  "data": {
    "isEmpty": false
  }
}
WARN  [WARN] 17:44.998 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:45.485 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:45.712 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:46.81 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:46.326 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:46.529 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:46.883 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:47.244 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:47.412 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:47.730 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
LOG  2024-01-10 07:17:48.176-08:00 [DATASTORE / ApplicationStack_Synced] DataStore event received:
LOG  {
 "event": "outboxStatus",
 "data": {
   "isEmpty": true
 }
}

I am hopeful this error is quite reproducible on your end, but next step is I'm going to check the AppSync logs in CloudWatch and I'll report back on anything interesting I find there.

@danrivett
Copy link

Looking more into my error directly above, I think it's a separate issue to the one @dmitryusikriseapps reported, but a bug in its own right that I'd like to see fixed.

Regarding the original ReferenceError: Property 'err' doesn't exist error, we're still trying to narrow down a easy recreate, but we can recreate it when we have a function that invokes DataStore.save() on one model and then immediately invokes DataStore.delete() on another model to clear out some stale data.

I'm wondering if this issue is related to the React Native environment, and the Hermes engine in particular, as some initial searching indicates it may be.

Here is the where the error is coming from:

Not being able to reference err seems odd, and makes me think it may be Hermes specific?

We'll try and provide as simple a problem recreate as possible, as we'd really like to see any bug identified fixed, as this is totally blocking our migration to v6.

@danrivett
Copy link

danrivett commented Jan 11, 2024

@chrisbonifacio I've spent quite a bit of time trying to narrow down the problem recreation scenario to help recreate this bug in as simple as manner possible.

What I found is I could reliably recreate both the ReferenceError: Property 'err' doesn't exist error, as well as the DataStore - conflict trycatch [TypeError: WeakSet key must be an Object] error with just a simple change.

Note: I recreated this in a React Native application, and the former error in particular may only be recreatable there, I'm not sure.

I am hopeful the instructions below should allow you or others on the team (perhaps @iartemiev ?) to recreate these too with minimal effort. Both @dmitryusikriseapps and I are happy to help as currently this bug(s) is blocking us from migrating to v6 and we'd really like to do that.

Problem Recreation Scenarios

The code below works on a simple @model type (ContractorJobInfo) and operates in 2 primary different scenarios

  1. When a matching ContractorJobInfo doesn't already exist
    • In this case it first creates a new ContractorJobInfo object in DataStore, and then updates it.
  2. When a matching ContractorJobInfo does already exist
    • In this case it first updates the existing ContractorJobInfo object retrieved from DataStore, then updates it again.

These two scenarios are then repeated but a 5 second sleep is added between the two DataStore mutation operations above.

This gives 4 different scenaros so far.

Finally, I noticed that adding a Pipeline Resolver for the createContractorJobInfo and updateContractorJobInfo which sets a field itself causes different test results, and so the 4 different scenarios are repeated both with these pipeline resolvers active, and without.

This leads to 8 total test scenarios that I've tested and documented my findings.

Problem Recreation Code

const sleep = (ms: number): Promise<void> => {
  return new Promise(resolve => {
    setTimeout(resolve, ms);
  });
};

let jobInfo = (await DataStore.query(DataStoreModels.ContractorJobInfo, info => info.jobId.eq('dummy-id')))[0];

if (jobInfo == null) {
  console.log(`Creating ContractorJobInfo`);

  jobInfo = await DataStore.save(
    new DataStoreModels.ContractorJobInfo({
      contractorId: currentContractor.id,
      cognitoUserId: currentContractor.cognitoUserId,
      jobId: 'dummy-id',
    })
  );
} else {
  console.log(`Updating ContractorJobInfo initially`);

  jobInfo = await DataStore.save(
    DataStoreModels.ContractorJobInfo.copyOf(jobInfo, updated => {
      updated.contractorId = `changed-${dayjs().toISOString()}`;
    })
  );
}

// Some of the test scenarios sleep between DataStore operations, others don't
// This is to see how adding a delay (for outbox syncing) affects things

// await sleep(5000);

console.log(`Updating ContractorJobInfo`);

await DataStore.save(
  DataStoreModels.ContractorJobInfo.copyOf(jobInfo, updated => {
    updated.contractorId = 'overwritten';
  })
);

Pipeline Resolvers

As I mentioned above we have some additional custom PipelineResolvers for createContractorJobInfo and updateContractorJobInfo which add some audit fields keeping track of who last edited a model type as we have some administrators who can modify other people's ContractorJobInfo records and we want the server (rather than the client) to keep track of who is editing it. Each update is then logged separately to a CDC bucket so we can see who edited which version:

Mutation.createContractorJobWorkerInfo.preUpdate.1.req.vtl

This is simplified from what we have, but illustrates the important part.

## We don't allow audit fields to be set manually, we set them
$util.qr($context.args.input.remove("lastEdited"))

#if( $util.authType() == "User Pool Authorization" )
  #set( $currentUserId = $util.defaultIfNull($ctx.identity.claims.get("username"), $util.defaultIfNull($ctx.identity.claims.get("cognito:username"), null)) )
  #set( $now = $util.time.nowISO8601() )
  #set( $currentVersion = $util.defaultIfNull($context.args.input["_version"], 0) )

  $util.qr( $context.args.input.put("createdBy", $currentUserId) )

  ## The 'lastEdited' fields refer to the last time a type was created/updated by a human and not the BE
  ## This includes both a regular user who has access, as well as an internal admin user
  #set( $lastEdited = {
    "cognitoUserId": $currentUserId,
    "at": $now,
    "versionEdited": $currentVersion
  } )

  $util.qr( $context.args.input.put("lastEdited", $lastEdited) )
#end

Mutation.updateContractorJobWorkerInfo.preUpdate.1.req.vtl

Again this is simplified from what we have.

## We don't allow audit fields to be set manually, we set them
$util.qr($context.args.input.remove("lastEdited"))

#if( $util.authType() == "User Pool Authorization" )
  #set( $currentUserId = $util.defaultIfNull($ctx.identity.claims.get("username"), $util.defaultIfNull($ctx.identity.claims.get("cognito:username"), null)) )
  #set( $now = $util.time.nowISO8601() )
  #set( $currentVersion = $util.defaultIfNull($context.args.input["_version"], 0) )

  #set( $lastEdited = {
    "cognitoUserId": $currentUserId,
    "at": $now,
    "versionEdited": $currentVersion
  } )

  $util.qr( $context.args.input.put("lastEdited", $lastEdited) )
#end

I'll add the test results in the following comment as this comment is already very large.

@danrivett
Copy link

danrivett commented Jan 11, 2024

Problem Recreation Results

Based on the code above and 8 different test scenarios I've described, I can consistently recreate both the ReferenceError: Property 'err' doesn't exist error, as well as the DataStore - conflict trycatch [TypeError: WeakSet key must be an Object] error.

Though I'll state immediately that I could only recreate the ReferenceError: Property 'err' doesn't exist error when the custom Pipeline Resolver was active that updated a field itself.

So we'll start when custom Pipeline Resolvers are active:

1. With Custom Pipeline Resolvers Active

Note: The custom pipeline resolver must modify the data for it to recreate the results below.

1.1 With no sleep(5000):

1.1.1 When no existing record exists:

Result: No error is returned, but although the initial create works, I do not see the subsequent update written to DynamoDB.
The contractorId field is not set to overwritten and _version remains 1 from the initial create and isn't incremented to 2.

1.1.2 When an existing record exists:

This can be tested by repeating the same test again after 1.1.1 as there's now a matching ContractorJobInfo record persisted.

Result: No error is returned, but similar to 1.1.1, although the initial update works, I do not see the 2nd update written to DynamoDB.
The contractorId field is set to changed-... and is not set to overwritten, and _version is incremented to 2 not 3.

So far these test results don't show either of the two errors reported above, but they do highlight a new 3rd error - a silent missing update. Just as important if not more so (as it's a silent error).

1.2 With sleep(5000) in between:

1.2.1 When no existing record exists:

Result: The initial create succeeds, but the subsequent update fails with a single attempt and the error: ReferenceError: Property 'err' doesn't exist.

1.2.2 When an existing record exists:

Result: The initial update succeeds, but the subsequent update fails with 10 attempts all failing with: DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]

At this point the test scenarios above all reliably recreate both the originally reported errors, plus the new silent error. This should be all that's needed to recreate all of them, but for completeness I'll include the other test scenario results. Especially as it shows the difference not having a custom Pipeline Resolver that modifies the data itself.

2. With no Custom Pipeline Resolvers Active

2.1 With no sleep(5000):

2.1.1 When no existing record exists:

Result: Same result as 1.1.1.

2.1.2 When an existing record exists:

Result: Same result as 1.1.2.

2.2 With sleep(5000) in between:

2.2.1 When no existing record exists:

Result: The initial create succeeds, but the subsequent update fails with 10 attempts all failing with: DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]

2.2.2 When an existing record exists:

Result: Same result as 1.1.2.

@danrivett
Copy link

danrivett commented Jan 11, 2024

Problem Summary

From my problem recreation scenarios and results above, I observed a few things.

  1. If mutations on the same model record occurred back to back without any gap between, it appears possible for lost updates to occur.

    • See results for test scenarios 1.1.1 and 1.1.2 for examples.
  2. If mutations occur of the same type on the same model record (e.g. two update mutations) with a time gap between them (5 seconds in my case), then the DataStore - conflict trycatch [TypeError: WeakSet key must be an Object] error occurs

    • This is irrespective of whether there are custom pipeline resolvers modifying the data
    • See result for test scenario 1.2.2 as an example
  3. If mutations occur of a different type on the same model record (e.g. a create mutation, followed by an update mutation) with a time gap between them, but there is no custom Pipeline Resolver modifying the data, then the DataStore - conflict trycatch [TypeError: WeakSet key must be an Object] error occurs.

    • See result for test scenario 2.2.1 as an example
  4. Finally, if mutations occur of a different type on the same model record (e.g. a create mutation, followed by an update mutation) with a time gap between them, and there is a custom Pipeline Resolver modifying the data, then the ReferenceError: Property 'err' doesn't exist error occurs.

    • See result for test scenario 1.2.1 as an example
    • I wonder if a different error occurs in a non React Native environment as that error seems odd based on the location of it I referenced in a prevous comment. It looks like a different error occurs which then turns into ReferenceError: Property 'err' doesn't exist when run under Hermes.
  5. Incidentally, we actually noticed that for observation 4 above, it didn't actually need to be the same model record, or even the same model type, we had two different types with an update on one, followed by a delete on another, and we experienced ReferenceError: Property 'err' doesn't exist. And only the model updated (not deleted) had a Custom Pipeline Resolver.

That complicated the problem recreation steps, so for my recreation scenarios above I just used one model type, but it may be worth noting as it seems a Custom Pipeline resolver modifying data on the first mutation is ultimately what causes the 2nd mutation to fail with ReferenceError: Property 'err' doesn't exist. But I haven't confirmed that.

Next Steps

I've tried to provide as much information as I can in order for you to recreate these problems simply and reliably. They seem high priority to me given the errors. Certainly we cannot migrate to v6 until they are fixed.

If you can't recreate them, both @dmitryusikriseapps and I are happy to assist.

You may need to test in a React Native environment to recreate some of these issues.

We use Expo, and we had to build a custom Development Client through Expo to test with because v6 of the Amplify libraries aren't natively linked in Expo Go (see this Expo discussion: expo/expo/discussions/25586).

I'm hopeful experienced AWS DataStore developers such as @chrisbonifacio and @iartemiev can easily recreate this issue and then look for the root cause. That is where our experience really runs dry, but we're happy to assist as we can.

@cwomack cwomack added the DataStore Related to DataStore category label Jan 11, 2024
@chrisbonifacio
Copy link
Member

Hi @danrivett! Thank you for taking the time to deep dive into the issues you're experiencing and sharing your findings. I don't have an update just yet, but wanted to let you know we will be looking into it and what you've shared so far helps a ton!

@cwomack cwomack added investigating This issue is being investigated and removed pending-triage Issue is pending triage labels Jan 17, 2024
@nadetastic nadetastic added the React Native React Native related issue label Mar 4, 2024
@danrivett
Copy link

@chrisbonifacio it seems like your team has been very busy with other initiatives (gen 2 etc) so I've avoided asking for updates up until now to give some space and time whilst your team is working on other things, but this is still blocking my team from upgrading to v6, and so is becoming higher in priority for us.

Is there anything we can do to assist in troubleshooting or investigating this, as we'd really love to see this investigated and any necessary fixes made.

@chrisbonifacio chrisbonifacio removed their assignment May 20, 2024
@danrivett
Copy link

I'd definitely appreciate an update from AWS of some sort.

The radio silence is really disconcerting, as it's a blocker for us to migrate to v6.

We're willing to assist in recreation and root cause analysis if needed, but we can't solve the problem entirely on our own, and hope AWS Support will engage to assist in resolving this.

@danrivett
Copy link

I'm not sure who's leading Amplify DataStore support, but I'm very happy to work directly with them either through this ticket or preferably through an AWS Chime meeting to discuss and work out a plan of attack.

@chrisbonifacio chrisbonifacio added the question General question label Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DataStore Related to DataStore category investigating This issue is being investigated question General question React Native React Native related issue
Projects
None yet
Development

No branches or pull requests

5 participants