Exception: A Parser can not resolve classes when using `$ref` if references are nested #300

sonali686 · 2021-01-13T13:00:47Z

Describe the bug
Parser can not resolve classes when using $ref if the path is nested even though the reference is a valid json pointer.

File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\__main__.py", line 281, in main
    generate(
  File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\__init__.py", line 259, in generate
    results = parser.parse()
  File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\parser\base.py", line 332, in parse
    _, sorted_data_models, require_update_action_models = sort_data_models(
  File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\parser\base.py", line 110, in sort_data_models
    return sort_data_models(
  File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\parser\base.py", line 110, in sort_data_models
    return sort_data_models(
  File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\parser\base.py", line 110, in sort_data_models
    return sort_data_models(
  [Previous line repeated 97 more times]
  File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\parser\base.py", line 158, in sort_data_models
    raise Exception(f'A Parser can not resolve classes: {unresolved_classes}.')
Exception: A Parser can not resolve classes: [class: Person references: {'Dog', 'Cat'}]

To Reproduce

Example schema:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Person",
    "type": "object",
    "properties": {
        "name": {
            "title": "name",
            "type": "string"
        },
        "pet": {
            "title": "pet",
            "type": "object",
            "oneOf": [
                {
                    "$ref": "#/definitions/Pets/Cat"
                },
                {
                    "$ref": "#/definitions/Pets/Dog"
                }
            ]
        }
    },
    "definitions": {
        "Pets": {
            "Cat": {
                "title": "Cat",
                "type": "object",
                "required": [
                    "pet_type",
                    "hunts",
                    "age"
                ],
                "properties": {
                    "pet_type": {
                        "enum": [
                            "Cat"
                        ]
                    },
                    "hunts": {
                        "type": "boolean"
                    },
                    "age": {
                        "type": "string"
                    }
                }
            },
            "Dog": {
                "title": "Dog",
                "type": "object",
                "required": [
                    "pet_type",
                    "bark",
                    "breed"
                ],
                "properties": {
                    "pet_type": {
                        "enum": [
                            "Dog"
                        ]
                    },
                    "bark": {
                        "type": "boolean"
                    },
                    "breed": {
                        "type": "string"
                    }
                }
            }
        }
    },
    "additionalProperties": false
}

Used commandline:

$ datamodel-codegen --input pets.json --input-file-type=jsonschema --output pets.py

Expected behavior
I was expecting following things which is working when I do not encapsulate "Cat" and "Dogs" inside "Pet".

Create the classes "Cat" and "Dog"
Add "pet" attribute with Union of "Cat" and "Dog"

Version:

windows
Python version: 3.9.0
datamodel-code-generator version:0.6.18

Additional context
I tried to pull the code and run this example but I am getting the same error. I have few large schemas which are encapsulating references for readability. After editing the schema according to datamodel-code-generator, it works wonderfully 👍 . Thanks for this module.

The text was updated successfully, but these errors were encountered:

koxudaxi · 2021-01-18T17:01:24Z

@sonali686
I'm sorry. I have overlooked a notification for this issue.

OK, I understand why Cat and Dog models are not generated.
This parse finds JSONSchema objects in each path.
But,definitions/Pets is not object.

I just decided on how to implement it.
If $ref is a JSON pointer and It is not parsed, the parser parses it after all models parsed.

I will support it. please wait...

koxudaxi · 2021-01-19T19:10:59Z

@sonali686
I fixed it!! And I have released a new version as 0.6.22.

MikeZaharov · 2021-01-22T08:40:15Z

Hi! I have the similar problem for the following case:
test.json

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "test.json",
  "description": "test",
  "type": "object",
  "required": [
    "test_id",
    "test_ip",
    "result"
  ],
  "properties": {
    "test_id": {
      "type": "string",
      "description": "test ID"
    },
    "test_ip": {
      "$ref": "base_test.json#/definitions/first"
    }
  }
}

base_test.json:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "base_test.json",
  "description": "test",
  "type": "object",
  "definitions": {
    "first": {
      "$ref": "#/definitions/second"
    },
    "second": {
      "type": "string"
    }
  }
}

Used commandline:

datamodel-codegen --input test.json --input-file-type=jsonschema --output model.py

Error:

raise Exception(f'A Parser can not resolve classes: {unresolved_classes}.')
Exception: A Parser can not resolve classes: [class: TestTestTestEvent references: {'First'}], [class: First references: {'Second'}].

The issue can be solved by editing ref link:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "base_test.json",
  "description": "test",
  "type": "object",
  "definitions": {
    "first": {
      "$ref": "base_test.json#/definitions/second"
    },
    "second": {
      "type": "string"
    }
  }
}

But i don't sure it's the right solution

Version:

OS: Ubuntu 16.04
Python version: 3.7
datamodel-code-generator version: 0.6.23

koxudaxi · 2021-01-24T16:11:56Z

Hi @MikeZaharov,
Thank you for the bug report.
I just fixed it.
And, I have released a new version 0.6.25
I tested your schemas with the version.
model.py

# generated by datamodel-codegen:
#   filename:  test.json
#   timestamp: 2021-01-24T16:08:55+00:00

from __future__ import annotations

from pydantic import BaseModel, Field


class Second(BaseModel):
    __root__: str


class First(BaseModel):
    __root__: Second


class Model(BaseModel):
    test_id: str = Field(..., description='test ID')
    test_ip: First

MikeZaharov · 2021-01-25T18:57:10Z

@koxudaxi
I've just tested and it works correctly.
Thank you) This library is awesome!)

sonali686 · 2021-02-24T12:04:04Z

@koxudaxi
I am using latest version (0.8.2) of the library and delighted to see many new options. Thanks for the awesome library once again :)

I came across one more scenario where library fails to resolve classes while using $ref.

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Person",
    "type": "object",
    "properties": {
        "name": {
            "title": "name",
            "type": "string"
        },
        "pet": {
            "title": "pet",
            "type": "object",
            "oneOf": [
                {
                    "$ref": "#/definitions/Pets/Cat"
                },
                {
                    "$ref": "#/definitions/Pets/Dog"
                }
            ]
        }
    },
    "definitions": {
        "CatBreed":{
            "C1":
            {
                "title":"C1",
                "type": "object",
                "properties":
                {
                    "hunts": {
                        "type": "boolean"
                    },
                    "age": {
                        "type": "string"
                    }
                }
            },
            "C2":
            {
                "title":"C2",
                "type": "object",
                "properties":
                {
                    "hunts": {
                        "type": "boolean"
                    },
                    "age": {
                        "type": "string"
                    }
                }
            }
        },
        "DogBreed":{
            "D1":
            {
                "title":"D1",
                "type": "object",
                "properties":
                {
                    "bark": {
                        "type": "boolean"
                    },
                    "age": {
                        "type": "string"
                    }
                }
            },
            "D2":
            {
                "title":"D2",
                "type": "object",
                "properties":
                {
                    "hunts": {
                        "type": "boolean"
                    },
                    "age": {
                        "type": "string"
                    }
                }
            }
        },
        "Pets": {
            "Cat": {
                "title": "Cat",
                "type": "object",
                "required": [
                    "pet_type",
                    "age"
                ],
                "properties": {
                    "pet_type": {
                        "enum": [
                            "Cat"
                        ]
                    },
                    "breed": {
                        "title": "breed",
                        "type": "object",
                        "oneOf": [
                            {
                                "$ref": "#/definitions/CatBreed/C1"
                            },
                            {
                                "$ref": "#/definitions/CatBreed/C2"
                            }
                        ]
                    }
                }
            },
            "Dog": {
                "title": "Dog",
                "type": "object",
                "required": [
                    "pet_type",
                    "breed"
                ],
                "properties": {
                    "pet_type": {
                        "enum": [
                            "Dog"
                        ]
                    },
                    "breed": {
                        "title": "breed",
                        "type": "string",
                        "oneOf": [
                            {
                                "$ref": "#/definitions/DogBreed/D1"
                            },
                            {
                                "$ref": "#/definitions/DogBreed/D2"
                            }
                        ]
                    }
                }
            }
        }
    },
    "additionalProperties": false
}

Is it possible to support this scenario as well? Is it possible to resolve any reference if it is a valid json pointer? Currently I am getting same exception as mentioned in my original post.

koxudaxi · 2021-02-24T17:52:51Z

@sonali686
Wow, I don't expect the pattern.
The code-generator resolves JsonPointer after parsing all non-JsonPointer objects.
Your example has JsonPointer is in JsonPointer.
It means the nested JsonPointer is not resolved.

I have fixed the code with my hands instantly.
It can generate it :)
I will push the complete fixed version tomorrow.
Thank you.

I put generated models.
Did you expect the models?

# generated by datamodel-codegen:
#   filename:  person.json
#   timestamp: 2019-07-26T00:00:00+00:00

from __future__ import annotations

from enum import Enum
from typing import Any, Optional, Union

from pydantic import BaseModel, Field


class CatBreed(BaseModel):
    __root__: Any


class DogBreed(BaseModel):
    __root__: Any


class Pets(BaseModel):
    __root__: Any


class PetType(Enum):
    Cat = 'Cat'


class PetType1(Enum):
    Dog = 'Dog'


class C1(BaseModel):
    hunts: Optional[bool] = None
    age: Optional[str] = None


class C2(BaseModel):
    hunts: Optional[bool] = None
    age: Optional[str] = None


class D1(BaseModel):
    bark: Optional[bool] = None
    age: Optional[str] = None


class D2(BaseModel):
    hunts: Optional[bool] = None
    age: Optional[str] = None


class Cat(BaseModel):
    pet_type: PetType
    breed: Optional[Union[C1, C2]] = Field(None, title='breed')


class Dog(BaseModel):
    pet_type: PetType1
    breed: Union[D1, D2] = Field(..., title='breed')


class Person(BaseModel):
    name: Optional[str] = Field(None, title='name')
    pet: Optional[Union[Cat, Dog]] = Field(None, title='pet')

sonali686 · 2021-02-25T06:29:29Z

@sonali686
Wow, I don't expect the pattern.
The code-generator resolves JsonPointer after parsing all non-JsonPointer objects.
Your example has JsonPointer is in JsonPointer.
It means the nested JsonPointer is not resolved.

I have fixed the code with my hands instantly.
It can generate it :)
I will push the complete fixed version tomorrow.
Thank you.

I put generated models.
Did you expect the models?

# generated by datamodel-codegen:
#   filename:  person.json
#   timestamp: 2019-07-26T00:00:00+00:00

from __future__ import annotations

from enum import Enum
from typing import Any, Optional, Union

from pydantic import BaseModel, Field


class CatBreed(BaseModel):
    __root__: Any


class DogBreed(BaseModel):
    __root__: Any


class Pets(BaseModel):
    __root__: Any


class PetType(Enum):
    Cat = 'Cat'


class PetType1(Enum):
    Dog = 'Dog'


class C1(BaseModel):
    hunts: Optional[bool] = None
    age: Optional[str] = None


class C2(BaseModel):
    hunts: Optional[bool] = None
    age: Optional[str] = None


class D1(BaseModel):
    bark: Optional[bool] = None
    age: Optional[str] = None


class D2(BaseModel):
    hunts: Optional[bool] = None
    age: Optional[str] = None


class Cat(BaseModel):
    pet_type: PetType
    breed: Optional[Union[C1, C2]] = Field(None, title='breed')


class Dog(BaseModel):
    pet_type: PetType1
    breed: Union[D1, D2] = Field(..., title='breed')


class Person(BaseModel):
    name: Optional[str] = Field(None, title='name')
    pet: Optional[Union[Cat, Dog]] = Field(None, title='pet')

@koxudaxi

Yes I expected these models. Thanks for providing the solution so fast. 👍

koxudaxi · 2021-02-25T14:48:13Z

@sonali686
Thank you for your response!!

I have released a new version as 0.8.3!!

koxudaxi mentioned this issue Jan 19, 2021

Support json pointer #311

Merged

koxudaxi mentioned this issue Jan 24, 2021

Improve resolving references #318

Merged

koxudaxi added the released label Jan 24, 2021

koxudaxi closed this as completed Feb 13, 2021

koxudaxi mentioned this issue Feb 25, 2021

Support nested json_pointer #352

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exception: A Parser can not resolve classes when using `$ref` if references are nested #300

Exception: A Parser can not resolve classes when using `$ref` if references are nested #300

sonali686 commented Jan 13, 2021

koxudaxi commented Jan 18, 2021

koxudaxi commented Jan 19, 2021

MikeZaharov commented Jan 22, 2021

koxudaxi commented Jan 24, 2021

MikeZaharov commented Jan 25, 2021

sonali686 commented Feb 24, 2021 •

edited

koxudaxi commented Feb 24, 2021

sonali686 commented Feb 25, 2021

koxudaxi commented Feb 25, 2021

Exception: A Parser can not resolve classes when using $ref if references are nested #300

Exception: A Parser can not resolve classes when using $ref if references are nested #300

Comments

sonali686 commented Jan 13, 2021

koxudaxi commented Jan 18, 2021

koxudaxi commented Jan 19, 2021

MikeZaharov commented Jan 22, 2021

koxudaxi commented Jan 24, 2021

MikeZaharov commented Jan 25, 2021

sonali686 commented Feb 24, 2021 • edited

koxudaxi commented Feb 24, 2021

sonali686 commented Feb 25, 2021

koxudaxi commented Feb 25, 2021

Exception: A Parser can not resolve classes when using `$ref` if references are nested #300

Exception: A Parser can not resolve classes when using `$ref` if references are nested #300

sonali686 commented Feb 24, 2021 •

edited