Include custom annotations in generated schemas / code

[simontaurus]

Is your feature request related to a problem? Please describe.
We are currently working on a link schema repo https://opensemantic.world/ combining json-schema and json-ld, similar to #474
On the way we recognized that linkml (great work!) already addresses a lot of problems we have identified, especially the bridges to OWL and SHACL.
However, since we focus both on programmatical and graphical interfaces would also need some custom annotations
for custom vocabular e. g. used by https://github.com/json-editor/json-editor to generate html forms.

Describe the solution you'd like
Option to transfer

• a) existing annotations (like range even when inlined_as_list: false)
• b) custom annotation like hidden: true for autogenerated fields in forms
  into jsonschema and pydantic classes

  classes:
  Person:
  Container:
    attributes:
      id:
        option:
          hidden: true
      persons:
        multivalued: true
        inlined_as_list: true
        range: Person

{
    "$defs": {
        "Container": {
            "description": "",
            "properties": {
                "id": {
                    "options": {
                        "hidden": "true"
                    },
                    "type": "string"
                "persons": {
                    "items": {
                        "type": "string"
                    },
                    "type": "array",
                    "range": "#/$defs/Person"
                }
            },
            "title": "Container",
            "type": "object"
        },
        "Person": {
        }
    }
}

class Person:
    pass

class Container(ConfiguredBaseModel):
    id: str
    persons: Optional[List[str]] = Field(default_factory=list, range=Person, options={'hidden': True})

Note: the extra parameter for Field() is also generated by https://github.com/koxudaxi/datamodel-code-generator

Maybe utilizing linkml annotations and dump the value as dict in the json-schema property and the pydantic Field if the generator was called with --include-annotations=True would be a solution for b)

How important is this feature? Select from the options below:
• Important - it's a blocker and can't do work without it (but of course I understand if this is out-of-scope)

When will use cases depending on this become relevant? Select from the options below:
• Mid-term - 2-4 months

Additional context
Related:

• Keyword to indicate/restrict the schema of a target document of a property of type URI/IRI  json-schema-org/json-schema-vocabularies#55

[cmungall]

This is in scope and would be a useful feature. We want to think carefully about how best to do this. See also #1830

I think using options in pydantic makes sense. It would be nice if there were an equivalent for dataclasses. @pkalita-lbl does the jsonschema approach above seem reasonable

[pkalita-lbl]

I think encoding the extra information in annotations along with an optional generator flag like --inject-annotations could work well for the JSON Schema generator. We'd just want to make sure we're careful to warn the user about edge cases like if the annotation would overwrite a JSON Schema keyword that was produced by the existing generation logic. Such as:

slots:
  age:
    minimum_value: 0  # this will generate `minimum: 0` in JSON Schema
    annotations:
      hidden: true
      minimum: 10  # uh oh!

[cmungall]

Do we also have to worry about the scenario where a future version of json-schema introduces a keyword annotations?

[simontaurus]

Following @pkalita-lbl approach, annotations would not collide with json-schema keywords since it's not exported (but the keywords / subobjects below)
So running

slots:
  age:
    minimum_value: 0  # this will generate `minimum: 0` in JSON Schema
    annotations:
      options: 
        hidden: true
      minimum: 10  # uh oh!

with --inject-annotations would lead to

{
  "type": "object",
  "properties": {
                "age": {
                    "type": "int",
                    "minimum": 10
                    "options": {
                        "hidden": true
                    }
               }
}

with minimum overwritten by annotation. I think we can put the responsibility on the user and may introduce in addition --inject-annotation-prefix 'x-', resulting in

{
  "type": "object",
  "properties": {
                "age": {
                    "type": "int",
                    "minimum": 0
                    "x-options": {
                        "hidden": true
                    },
                    "x-minimum": 10
               }
}

or even --inject-annotation-prefix-on-conflict, resulting in

{
  "type": "object",
  "properties": {
                "age": {
                    "type": "int",
                    "minimum": 0
                    "options": {
                        "hidden": true
                    },
                    "x-minimum": 10
               }
}

[pkalita-lbl]

I think we can put the responsibility on the user

I agree. I just wanted to flag it as something we should be clear on and communicate to the user (through documentation, runtime warnings, etc) so that no one is surprised.

[jsheunis]

I'd like to register my interest in this feature as well, in relation to the automatic generation of user interfaces from schemas (similar to @simontaurus's use case). I'm focusing specifically on SHACL. I've adapted shaclgen.py in my local clone, enough to be able to see custom annotations on slots (directly, and slots with custom types as ranges, where the types have annotations) flow through to the exported SHACL. Perhaps my comments provide some more meat for the use case:

1. I am following the design at https://datashapes.org/forms.html quite closely, i.e. using the DASH vocabulary to define constraints for how specific fields are to be edited/viewed.
2. I am hoping this is just a result of my ignorance, but I am not sure how I could coerce the annotations into specific types on the SHACL side. E.g. if I have the following schema:

   id: https://example.org/test-schema
   name: myschema
   
   prefixes:
     dash: http://datashapes.org/dash#
     myschema: https://example.org/test-schema/
   
   default_prefix: myschema
   
   emit_prefixes:
     - dash
   
   imports: https://w3id.org/linkml/types
   
   slots:
     my_attr:
       range: string
       annotations:
         dash:singleLine: true
   
   classes:
     MyClass:
       abstract: true
       slots:
         - my_attr
   

   I want the SHACL to be:

   @prefix myschema: <https://example.org/test-schema/> .
   @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
   @prefix sh: <http://www.w3.org/ns/shacl#> .
   @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
   
   myschema:MyClass a sh:NodeShape ;
       sh:closed false ;
       sh:ignoredProperties ( rdf:type ) ;
       sh:property [ sh:datatype xsd:string ;
               dash:singleLine true ;
               sh:maxCount 1 ;
               sh:nodeKind sh:Literal ;
               sh:order 0 ;
               sh:path myschema:my_attr ] ;
       sh:targetClass myschema:MyClass .
   

   i.e. the annotation tag dash:singleLine should be recognized as a CURIE and the annotation value true should be recognized as xsd:boolean, i.e. not string literals. I don't have ideas yet about how to achieve this.
3. I encountered a scenario where there could be conflicts between annotations of a slot and annotations of a custom type which is used as the range of that same slot. E.g.:

   types:
     NameString:
       typeof: string
       uri: myschema:NameString
       annotations:
         dash:singleLine: false
   
   slots:
     my_attr:
       range: NameString
       annotations:
         dash:singleLine: true
   

   Here, the question is whether the resulting property shape of my_attr in SHACL should have dash:singleLine true ; or dash:singleLine false ;. I thought it's worth mentioning such a scenario and to consider how generator code would deal with this if needed.

If anyone has some more insight to share about these challenges I'm facing, I would love to hear them! I'm happy to create a PR to address shaclgen improvements ito annotations, once the uncertainties are gone.

Disclaimers:

• I have used LinkML for some months now, but I'm almost fully ignorant to the code internals
• I haven't looked into the base Generators class that something like shaclgen inherits from, so I'm not sure if some of the functionality in my local patches would be general enough to be applied to the base class rather than in shaclgen specifically.

[sneakers-the-rat]

saw this when searching for related issues for a PR. this is done for pydanticgen at least via the metadata_mode property that can include all schema metadata in generated models, including annotations.