Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nested json is transformed to string when used “$type” key in object #21454

Closed
tomasz-py opened this issue Oct 11, 2023 · 9 comments · Fixed by #22850
Closed

Nested json is transformed to string when used “$type” key in object #21454

tomasz-py opened this issue Oct 11, 2023 · 9 comments · Fixed by #22850
Assignees
Labels
bug/2-confirmed Bug has been reproduced and confirmed. kind/regression A reported bug in functionality that used to work before. team/client Issue for team Client. topic: json array topic: Json Scalar type `Json`
Milestone

Comments

@tomasz-py
Copy link

Bug description

Json value which has array of objects where one of keys is "$type" is transformed to string

const data = await prisma.test.create({ data: { value: { options: [{ $type: "z" }, { $type: 1 }], }, }, });

data.value will looks like:
{ options: [ '{"$type":"z"}', '{"$type":1}' ] }

instead of
{ options: [ {'$type': 'z'}, {'$type': 1 } ] }

the same situation is for object in object:
const data = await prisma.test.create({ data: { value: { options: { $type: "test", }, }, }, });

data.value will contain options as string instead of json
{ options: '{"$type":"test"}' }

Additional info:
its stored properly if object key is $test, so its only breaks when key is "$type"

How to reproduce

  1. Use the following schema
  2. Generate the client and run the following script
  3. Check value in database

Expected behavior

Prisma should store array of jsons as jsons instead of transforming them into string

Prisma information

model Test {
  id  Int  @id @default(autoincrement())
  value  Json?  @db.JsonB
}
import { PrismaClient } from "@prisma/client";

async function main() {
  const prisma = new PrismaClient();

  const data = await prisma.test.create({
    data: {
       value: {
         options: [{ $type: "z" }, { $type: 1 }],
        },
    },
  });

 const otherData =  await prisma.test.create({
    data: {
        value: {
          options: {
            $type: "test",
          },
        },
    },
  });

  console.log(data.value); // { options: [ '{"$type":"z"}', '{"$type":1}' ] }
 
  console.log(otherData.value) // { options: '{"$type":"test"}' }

  prisma.$disconnect();
}

main();

Environment & setup

  • OS: Windows 11
  • Database: PostgreSQL
  • Node.js version: v20.5.1

Prisma Version

prisma                  : 5.1.1
@prisma/client          : 5.1.1
Current platform        : windows
Query Engine (Node-API) : libquery-engine 6a3747c37ff169c90047725a05a6ef02e32ac97e (at node_modules\@prisma\engines\query_engine-windows.dll.node)
Schema Engine           : schema-engine-cli 6a3747c37ff169c90047725a05a6ef02e32ac97e (at node_modules\@prisma\engines\schema-engine-windows.exe)
Schema Wasm             : @prisma/prisma-schema-wasm 5.1.1-1.6a3747c37ff169c90047725a05a6ef02e32ac97e
Default Engines Hash    : 6a3747c37ff169c90047725a05a6ef02e32ac97e
Studio                  : 0.492.0
@tomasz-py tomasz-py added the kind/bug A reported bug. label Oct 11, 2023
@apolanc apolanc added topic: Json Scalar type `Json` team/client Issue for team Client. topic: json array bug/1-unconfirmed Bug should have enough information for reproduction, but confirmation has not happened yet. labels Oct 13, 2023
@JosephHalter
Copy link

Thank you, I thought that I was getting crazy but indeed it's easy to reproduce, just try saving an array of objects with $type key in a json column and you'll see that the objects are serialized as strings instead, making it impossible to query inside or even read back without having to use JSON.parse recursively.

@tomasz-py
Copy link
Author

Thanks for confirmation.
Its critical bug for us because we have migrated to Prisma now and we found out that it will destroy our old data in database.

@janpio janpio added kind/regression A reported bug in functionality that used to work before. and removed kind/bug A reported bug. labels Oct 18, 2023
@niba
Copy link

niba commented Nov 10, 2023

We are having the same issue. We migrated to Prisma and we can't fetch data from one of our tables because when you fetch data with a structure containing "$type" then it explodes with error: Unknown tagged value.
I see that it is probably because of jsonProtocol implementation.

Do you have any workaround for this bug? @janpio

This issue is pretty critical for us, as there's no easy fix on our side.

@karanssj4
Copy link

can confirm this bug in prisma 5.1.1
any update on a workaround or where to look for to solve this?
we are not able to accept json from users because of this.

@karanssj4
Copy link

found a workaround.
set toJSON on the prototype of the object you want to insert.
This shouldn't have any problems for puse JS objects. maybe someone from the prisma team should confirm this? @janpio

change from

 const valueToInsert =  {
   value: {
      options: [{ $type: "z" }, { $type: 1 }],
    },
  }
};

Object.setPrototypeOf(valueToInsert, {
  toJSON() { return valueToInsert }
});

const data = await prisma.test.create({
  data: valueToInsert
});

@niba
Copy link

niba commented Nov 29, 2023

@karanssj4
nice, I tried to use toJSON but it didn't work in my case. I need to check again

another problem is that you can't fetch this kind of data from database because you get exception error :<
try to use findMany etc

$type is used in their implementation of JSONProtocol somewhere in rust engine and it goes crazy when you have key like that

I don't understand how this bug is not high priority right now

@karanssj4
Copy link

@niba i've not seen this issue while reading back from db, only while inserting it. we used raw queries for insrting and prisma crud methods for reading, it worked

can you share the read opearation that messes it up for you?

@SevInf SevInf self-assigned this Jan 19, 2024
SevInf added a commit that referenced this issue Jan 19, 2024
Reproduction for $type key in JSON issue. It works correctly if $type is
a top level key in JSON field, but fails if it is nested.
@SevInf
Copy link
Contributor

SevInf commented Jan 19, 2024

Reproduced the issue in #22714

@SevInf SevInf added bug/2-confirmed Bug has been reproduced and confirmed. and removed bug/1-unconfirmed Bug should have enough information for reproduction, but confirmation has not happened yet. labels Jan 19, 2024
@SevInf SevInf self-assigned this Jan 23, 2024
SevInf added a commit to prisma/prisma-engines that referenced this issue Jan 23, 2024
In JSON protocol, we use `$type` key as a special marker for the types
that can not be represented in JSON natively. We did not want to make
`$type` identifier reserved, so we took some precautions: if during
encoding, at any point, `$type` key would be seen in user's input we
encode whole object as `{"$type": "Json", "value": [Serialized object] }`.

Engine handled those cases correctly, when `$type: Json` is encountered
at the top level, but not when it is nested. In that case,
`ArgumentValue`-to-`JSON` serialization just picked `value` key and
saved it as a string.

Fixed by moving Json value into it's own kind of `ArgumentValue`, that
are serialized to json as-is.

Fix prisma/prisma#21454
SevInf added a commit to prisma/prisma-engines that referenced this issue Jan 24, 2024
In JSON protocol, we use `$type` key as a special marker for the types
that can not be represented in JSON natively. We did not want to make
`$type` identifier reserved, so we took some precautions: if during
encoding, at any point, `$type` key would be seen in user's input we
encode whole object as `{"$type": "Json", "value": [Serialized object] }`.

Engine handled those cases correctly, when `$type: Json` is encountered
at the top level, but not when it is nested. In that case,
`ArgumentValue`-to-`JSON` serialization just picked `value` key and
saved it as a string.

Fixed by moving Json value into it's own kind of `ArgumentValue`, that
are serialized to json as-is.

Fix prisma/prisma#21454
SevInf added a commit to prisma/prisma-engines that referenced this issue Jan 24, 2024
Introduce another special value to JSON protocol, `"$type": "Raw"`.
When encountered, no other nested `$type` keys would be interpreted as
special and will be written to DB as is. Main usecase is JSON column
values with user-provided `$type` keys.

This is an alternative to #4668, that might look cleaner on the engine
side.

Part of the fix for prisma/prisma#21454, will require client adjustments
as well.
SevInf added a commit to prisma/prisma-engines that referenced this issue Jan 24, 2024
Introduce another special value to JSON protocol, `"$type": "Raw"`.
When encountered, no other nested `$type` keys would be interpreted as
special and will be written to DB as is. Main usecase is JSON column
values with user-provided `$type` keys.

This is an alternative to #4668, that might look cleaner on the engine
side.

Part of the fix for prisma/prisma#21454, will require client adjustments
as well.
SevInf added a commit to prisma/prisma-engines that referenced this issue Jan 24, 2024
Introduce another special value to JSON protocol, `"$type": "Raw"`.
When encountered, no other nested `$type` keys would be interpreted as
special and will be written to DB as is. Main usecase is JSON column
values with user-provided `$type` keys.

This is an alternative to #4668, that might look cleaner on the engine
side.

Part of the fix for prisma/prisma#21454, will require client adjustments
as well.
SevInf added a commit to prisma/prisma-engines that referenced this issue Jan 26, 2024
Introduce another special value to JSON protocol, `"$type": "Raw"`.
When encountered, no other nested `$type` keys would be interpreted as
special and will be written to DB as is. Main usecase is JSON column
values with user-provided `$type` keys.

This is an alternative to #4668, that might look cleaner on the engine
side.

Part of the fix for prisma/prisma#21454, will require client adjustments
as well.
SevInf added a commit to prisma/prisma-engines that referenced this issue Jan 26, 2024
In JSON protocol, we use `$type` key as a special marker for the types
that can not be represented in JSON natively. We did not want to make
`$type` identifier reserved, so we took some precautions: if during
encoding, at any point, `$type` key would be seen in user's input we
encode whole object as `{"$type": "Json", "value": [Serialized object] }`.

Engine handled those cases correctly, when `$type: Json` is encountered
at the top level, but not when it is nested. In that case,
`ArgumentValue`-to-`JSON` serialization just picked `value` key and
saved it as a string.

Fixed by moving Json value into it's own kind of `ArgumentValue`, that
are serialized to json as-is.

Fix prisma/prisma#21454
SevInf added a commit to prisma/prisma-engines that referenced this issue Jan 29, 2024
Introduce another special value to JSON protocol, `"$type": "Raw"`.
When encountered, no other nested `$type` keys would be interpreted as
special and will be written to DB as is. Main usecase is JSON column
values with user-provided `$type` keys.

This is an alternative to #4668, that might look cleaner on the engine
side.

Part of the fix for prisma/prisma#21454, will require client adjustments
as well.
SevInf added a commit that referenced this issue Jan 29, 2024
Works on top of prisma/prisma-engines#4670.
Adds `$type: Raw` special case to JSON protocol and replaces `$type:
Json` encoding with it.

See engine PR for detailed description of the problem.

Fix #21454
@Jolg42 Jolg42 added this to the 5.9.0 milestone Jan 29, 2024
SevInf added a commit to prisma/prisma-engines that referenced this issue Jan 29, 2024
* qe: Fix nested objects with `$type` key in JSON protocol

Introduce another special value to JSON protocol, `"$type": "Raw"`.
When encountered, no other nested `$type` keys would be interpreted as
special and will be written to DB as is. Main usecase is JSON column
values with user-provided `$type` keys.

This is an alternative to #4668, that might look cleaner on the engine
side.

Part of the fix for prisma/prisma#21454, will require client adjustments
as well.

* Fix & move the test
SevInf added a commit that referenced this issue Jan 29, 2024
Works on top of prisma/prisma-engines#4670.
Adds `$type: Raw` special case to JSON protocol and replaces `$type:
Json` encoding with it.

See engine PR for detailed description of the problem.

Fix #21454
SevInf added a commit that referenced this issue Jan 29, 2024
Works on top of prisma/prisma-engines#4670.
Adds `$type: Raw` special case to JSON protocol and replaces `$type:
Json` encoding with it.

See engine PR for detailed description of the problem.

Fix #21454
@SevInf
Copy link
Contributor

SevInf commented Jan 29, 2024

Hello @tomasz-py @karanssj4 @niba.
The issue is fixed and the fix will be released in a version 5.9.0 tomorrow.
If you want to test dev snapshot, you can try 5.9.0-dev.64 version. We don't recommend you use dev snapshots in production, but it is enough to verify that fix works for your case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug/2-confirmed Bug has been reproduced and confirmed. kind/regression A reported bug in functionality that used to work before. team/client Issue for team Client. topic: json array topic: Json Scalar type `Json`
Projects
None yet
8 participants