Organize help options

[howardj99]

Address #1539

• Organize help options into groups
• Auto-sort help options by name within group
• Add --capitalize-enum-members as alternative spelling to existing --capitalise-enum-members options

Help output with this change applied:

usage: 
  datamodel-codegen [options]

Generate Python data models from schema definitions or structured data

Options:
  --http-headers HTTP_HEADER [HTTP_HEADER ...]
                        Set headers in HTTP requests to the remote host.
                        (example: "Authorization: Basic dXNlcjpwYXNz")
  --http-ignore-tls     Disable verification of the remote host's TLS
                        certificate
  --input INPUT         Input file/directory (default: stdin)
  --input-file-type {auto,openapi,jsonschema,json,yaml,dict,csv}
                        Input file type (default: auto)
  --output OUTPUT       Output file (default: stdout)
  --output-model-type {pydantic.BaseModel,pydantic_v2.BaseModel,dataclasses.dataclass,typing.TypedDict}
                        Output model type (default: pydantic.BaseModel)
  --url URL             Input file URL. `--input` is ignored when `--url` is
                        used

Typing customization:
  --base-class BASE_CLASS
                        Base Class (default: pydantic.BaseModel)
  --enum-field-as-literal {all,one}
                        Parse enum field as literal. all: all enum field type
                        are Literal. one: field type is Literal when an enum
                        has only one possible value
  --field-constraints   Use field constraints and not con* annotations
  --set-default-enum-member
                        Set enum members as default values for enum field
  --strict-types {str,bytes,int,float,bool} [{str,bytes,int,float,bool} ...]
                        Use strict types
  --use-annotated       Use typing.Annotated for Field(). Also, `--field-
                        constraints` option will be enabled.
  --use-generic-container-types
                        Use generic container types for type hinting
                        (typing.Sequence, typing.Mapping). If `--use-standard-
                        collections` option is set, then import from
                        collections.abc instead of typing
  --use-non-positive-negative-number-constrained-types
                        Use the Non{Positive,Negative}{FloatInt} types instead
                        of the corresponding con* constrained types.
  --use-one-literal-as-default
                        Use one literal as default value for one literal field
  --use-standard-collections
                        Use standard collections for type hinting (list, dict)
  --use-subclass-enum   Define Enum class as subclass with field type when
                        enum has type (int, float, bytes, str)
  --use-union-operator  Use | operator for Union type (PEP 604).
  --use-unique-items-as-set
                        define field type as `set` when the field attribute
                        has `uniqueItems`

Field customization:
  --capitalise-enum-members, --capitalize-enum-members
                        Capitalize field names on enum
  --empty-enum-field-name EMPTY_ENUM_FIELD_NAME
                        Set field name when enum value is empty (default: `_`)
  --field-extra-keys FIELD_EXTRA_KEYS [FIELD_EXTRA_KEYS ...]
                        Add extra keys to field parameters
  --field-extra-keys-without-x-prefix FIELD_EXTRA_KEYS_WITHOUT_X_PREFIX [FIELD_EXTRA_KEYS_WITHOUT_X_PREFIX ...]
                        Add extra keys with `x-` prefix to field parameters.
                        The extra keys are stripped of the `x-` prefix.
  --field-include-all-keys
                        Add all keys to field parameters
  --force-optional      Force optional for required fields
  --original-field-name-delimiter ORIGINAL_FIELD_NAME_DELIMITER
                        Set delimiter to convert to snake case. This option
                        only can be used with --snake-case-field (default: `_`
                        )
  --remove-special-field-name-prefix
                        Remove field name prefix when first character can't be
                        used as Python field name
  --snake-case-field    Change camel-case field name to snake-case
  --special-field-name-prefix SPECIAL_FIELD_NAME_PREFIX
                        Set field name prefix when first character can't be
                        used as Python field name (default: `field`)
  --strip-default-none  Strip default None on fields
  --use-default         Use default value even if a field is required
  --use-default-kwarg   Use `default=` instead of a positional argument for
                        Fields that have default values.
  --use-field-description
                        Use schema description to populate field docstring

Model customization:
  --allow-extra-fields  Allow to pass extra fields, if this flag is not
                        passed, extra fields are forbidden.
  --allow-population-by-field-name
                        Allow population by field name
  --class-name CLASS_NAME
                        Set class name of root model
  --collapse-root-models
                        Models generated with a root-type field will be
                        mergedinto the models using that root-type model
  --disable-appending-item-suffix
                        Disable appending `Item` suffix to model name in an
                        array
  --disable-timestamp   Disable timestamp on file headers
  --enable-faux-immutability
                        Enable faux immutability
  --enable-version-header
                        Enable package version on file headers
  --keep-model-order    Keep generated models' order
  --reuse-model         Re-use models on the field when a module has the model
                        with the same content
  --target-python-version {3.6,3.7,3.8,3.9,3.10,3.11}
                        target python version (default: 3.7)
  --use-schema-description
                        Use schema description to populate class docstring
  --use-title-as-name   use titles as class names of models

Template customization:
  --aliases ALIASES     Alias mapping file
  --custom-file-header CUSTOM_FILE_HEADER
                        Custom file header
  --custom-file-header-path CUSTOM_FILE_HEADER_PATH
                        Custom file header file path
  --custom-template-dir CUSTOM_TEMPLATE_DIR
                        Custom template directory
  --encoding ENCODING   The encoding of input and output (default: UTF-8)
  --extra-template-data EXTRA_TEMPLATE_DATA
                        Extra template data
  --use-double-quotes   Model generated with double quotes. Single quotes or
                        your black config skip_string_normalization value will
                        be used without this option.
  --wrap-string-literal
                        Wrap string literal by using black `experimental-
                        string-processing` option (require black 20.8b0 or
                        later)

OpenAPI-only options:
  --openapi-scopes {schemas,paths,tags,parameters} [{schemas,paths,tags,parameters} ...]
                        Scopes of OpenAPI model generation (default: schemas)
  --strict-nullable     Treat default field as a non-nullable field (Only
                        OpenAPI)
  --use-operation-id-as-name
                        use operation id of OpenAPI as class names of models
  --validation          Enable validation (Only OpenAPI)

General options:
  --debug               show debug message
  --disable-warnings    disable warnings
  --version             show version
  -h, --help            show this help message and exit

[howardj99]

@koxudaxi any suggestions or concerns?

[codecov]

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (e9b6edf) 100.00% compared to head (e6b0b73) 100.00%.
Report is 711 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##            master     #1540     +/-   ##
===========================================
  Coverage   100.00%   100.00%             
===========================================
  Files           11        33     +22     
  Lines         1020      3621   +2601     
  Branches       201       841    +640     
===========================================
+ Hits          1020      3621   +2601     

Flag	Coverage Δ
unittests	99.64% <99.59%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
datamodel_code_generator/__init__.py	100.00% <100.00%> (ø)
datamodel_code_generator/__main__.py	100.00% <100.00%> (ø)
datamodel_code_generator/arguments.py	100.00% <100.00%> (ø)
datamodel_code_generator/format.py	100.00% <100.00%> (ø)
datamodel_code_generator/http.py	100.00% <100.00%> (ø)
datamodel_code_generator/imports.py	100.00% <100.00%> (ø)
datamodel_code_generator/model/__init__.py	100.00% <100.00%> (ø)
datamodel_code_generator/model/base.py	100.00% <100.00%> (ø)
datamodel_code_generator/model/dataclass.py	100.00% <100.00%> (ø)
datamodel_code_generator/model/enum.py	100.00% <100.00%> (ø)
... and 23 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[koxudaxi]

@howardj99
Thank you for creating the PR.
I prefer your changes.
I left a few comments.

[howardj99]

@koxudaxi Should be good to run the checks again whenever you get a chance

[howardj99]

@koxudaxi How would you feel about moving the group/argument definitions into a JSON/YAML file to be read in and used as parameters to arg_parser.add_argument()?

Something like

{
  "general": [
    {
      "flags": ["--debug"],
      "help": "show debug message",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--disable-warnings"],
      "help": "disable warnings",
      "action": "store_true",
      "default": null
    }
  ]
}

just as an example. Should make __main__.py almost 400 lines shorter

[koxudaxi]

@howardj99

How would you feel about moving the group/argument definitions into a JSON/YAML file to be read in and used as parameters to arg_parser.add_argument()?

It looks good. We lack type-checking and code auto-completion. Still, the benefits will be significant.

[howardj99]

@howardj99

How would you feel about moving the group/argument definitions into a JSON/YAML file to be read in and used as parameters to arg_parser.add_argument()?

It looks good. We lack type-checking and code auto-completion. Still, the benefits will be significant.

Looks like it would have to be a jinja template or something rather than a static JSON file, since some of the parameters are dynamically generated with list comprehensions or bare types like str.

Full JSON example (invalid)

{
  "base": [
    {
      "flags": ["--http-headers"],
      "nargs": "+",
      "metavar": "HTTP_HEADER",
      "help": "Set headers in HTTP requests to the remote host. (example: \"Authorization: Basic dXNlcjpwYXNz\")"
    },
    {
      "flags": ["--http-ignore-tls"],
      "help": "Disable verification of the remote host's TLS certificate",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--input-file-type"],
      "help": "Input file type (default: auto)",
      "choices": [i.value for i in InputFileType]
    },
    {
      "flags": ["--input"],
      "help": "Input file/directory (default: stdin)"
    },
    {
      "flags": ["--output-model-type"],
      "help": "Output model type (default: pydantic.BaseModel)",
      "choices": [i.value for i in DataModelType]
    },
    {
      "flags": ["--output"],
      "help": "Output file (default: stdout)"
    },
    {
      "flags": ["--url"],
      "help": "Input file URL. `--input` is ignored when `--url` is used"
    }
  ],
  "typing": [
    {
      "flags": ["--base-class"],
      "help": "Base Class (default: pydantic.BaseModel)",
      "type": str
    },
    {
      "flags": ["--enum-field-as-literal"],
      "help": "Parse enum field as literal. all: all enum field type are Literal. one: field type is Literal when an enum has only one possible value",
      "choices": [lt.value for lt in LiteralType],
      "default": null
    },
    {
      "flags": ["--field-constraints"],
      "help": "Use field constraints and not con* annotations",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--set-default-enum-member"],
      "help": "Set enum members as default values for enum field",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--strict-types"],
      "help": "Use strict types",
      "choices": [t.value for t in StrictTypes],
      "nargs": "+"
    },
    {
      "flags": ["--use-annotated"],
      "help": "Use typing.Annotated for Field(). Also, `--field-constraints` option will be enabled.",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--use-generic-container-types"],
      "help": "Use generic container types for type hinting (typing.Sequence, typing.Mapping). If `--use-standard-collections` option is set, then import from collections.abc instead of typing",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--use-non-positive-negative-number-constrained-types"],
      "help": "Use the Non{Positive,Negative}{FloatInt} types instead of the corresponding con* constrained types.",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--use-one-literal-as-default"],
      "help": "Use one literal as default value for one literal field",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--use-standard-collections"],
      "help": "Use standard collections for type hinting (list, dict)",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--use-subclass-enum"],
      "help": "Define Enum class as subclass with field type when enum has type (int, float, bytes, str)",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--use-union-operator"],
      "help": "Use | operator for Union type (PEP 604).",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--use-unique-items-as-set"],
      "help": "define field type as `set` when the field attribute has `uniqueItems`",
      "action": "store_true",
      "default": null
    }
  ],
  "field": [
    {
      "flags": ["--capitalise-enum-members", "--capitalize-enum-members"],
      "help": "Capitalize field names on enum",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--empty-enum-field-name"],
      "help": "Set field name when enum value is empty (default:  `_`)",
      "default": null
    },
    {
      "flags": ["--field-extra-keys-without-x-prefix"],
      "help": "Add extra keys with `x-` prefix to field parameters. The extra keys are stripped of the `x-` prefix.",
      "type": str,
      "nargs": "+"
    },
    {
      "flags": ["--field-extra-keys"],
      "help": "Add extra keys to field parameters",
      "type": str,
      "nargs": "+"
    },
    {
      "flags": ["--field-include-all-keys"],
      "help": "Add all keys to field parameters",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--force-optional"],
      "help": "Force optional for required fields",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--original-field-name-delimiter"],
      "help": "Set delimiter to convert to snake case. This option only can be used with --snake-case-field (default: `_` )",
      "default": null
    },
    {
      "flags": ["--remove-special-field-name-prefix"],
      "help": "Remove field name prefix when first character can't be used as Python field name",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--snake-case-field"],
      "help": "Change camel-case field name to snake-case",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--special-field-name-prefix"],
      "help": "Set field name prefix when first character can't be used as Python field name (default:  `field`)",
      "default": null
    },
    {
      "flags": ["--strip-default-none"],
      "help": "Strip default None on fields",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--use-default-kwarg"],
      "action": "store_true",
      "help": "Use `default=` instead of a positional argument for Fields that have default values.",
      "default": null
    },
    {
      "flags": ["--use-default"],
      "help": "Use default value even if a field is required",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--use-field-description"],
      "help": "Use schema description to populate field docstring",
      "action": "store_true",
      "default": null
    }
  ],
  "model": [
    {
      "flags": ["--allow-extra-fields"],
      "help": "Allow to pass extra fields, if this flag is not passed, extra fields are forbidden.",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--allow-population-by-field-name"],
      "help": "Allow population by field name",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--class-name"],
      "help": "Set class name of root model",
      "default": null
    },
    {
      "flags": ["--collapse-root-models"],
      "action": "store_true",
      "default": null,
      "help": "Models generated with a root-type field will be mergedinto the models using that root-type model"
    },
    {
      "flags": ["--disable-appending-item-suffix"],
      "help": "Disable appending `Item` suffix to model name in an array",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--disable-timestamp"],
      "help": "Disable timestamp on file headers",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--enable-faux-immutability"],
      "help": "Enable faux immutability",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--enable-version-header"],
      "help": "Enable package version on file headers",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--keep-model-order"],
      "help": "Keep generated models' order",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--reuse-model"],
      "help": "Re-use models on the field when a module has the model with the same content",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--target-python-version"],
      "help": "target python version (default: 3.7)",
      "choices": [v.value for v in PythonVersion]
    },
    {
      "flags": ["--use-schema-description"],
      "help": "Use schema description to populate class docstring",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--use-title-as-name"],
      "help": "use titles as class names of models",
      "action": "store_true",
      "default": null
    }
  ],
  "template": [
    {
      "flags": ["--aliases"],
      "help": "Alias mapping file",
      "type": FileType("rt")
    },
    {
      "flags": ["--custom-file-header-path"],
      "help": "Custom file header file path",
      "default": null,
      "type": str
    },
    {
      "flags": ["--custom-file-header"],
      "help": "Custom file header",
      "type": str,
      "default": null
    },
    {
      "flags": ["--custom-template-dir"],
      "help": "Custom template directory",
      "type": str
    },
    {
      "flags": ["--encoding"],
      "help": f"The encoding of input and output (default: {DEFAULT_ENCODING})",
      "default": null
    },
    {
      "flags": ["--extra-template-data"],
      "help": "Extra template data",
      "type": FileType("rt")
    },
    {
      "flags": ["--use-double-quotes"],
      "action": "store_true",
      "default": null,
      "help": "Model generated with double quotes. Single quotes or your black config skip_string_normalization value will be used without this option."
    },
    {
      "flags": ["--wrap-string-literal"],
      "help": "Wrap string literal by using black `experimental-string-processing` option (require black 20.8b0 or later)",
      "action": "store_true",
      "default": null
    }
  ],
  "openapi": [
    {
      "flags": ["--openapi-scopes"],
      "help": "Scopes of OpenAPI model generation (default: schemas)",
      "choices": [o.value for o in OpenAPIScope],
      "nargs": "+",
      "default": null
    },
    {
      "flags": ["--strict-nullable"],
      "help": "Treat default field as a non-nullable field (Only OpenAPI)",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--use-operation-id-as-name"],
      "help": "use operation id of OpenAPI as class names of models",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--validation"],
      "help": "Enable validation (Only OpenAPI)",
      "action": "store_true",
      "default": null
    }
  ],
  "general": [
    {
      "flags": ["--debug"],
      "help": "show debug message",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--disable-warnings"],
      "help": "disable warnings",
      "action": "store_true",
      "default": null
    },
    {
      "flags": ["--no-color"],
      "help": "disable colorized output",
      "action": "store_true"
    },
    {
      "flags": ["--version"],
      "help": "show version",
      "action": "store_true"
    },
    {
      "flags": ["-h", "--help"],
      "action": "help",
      "default": "==SUPPRESS==",
      "help": "show this help message and exit"
    }
  ]
}

[howardj99]

@koxudaxi instead of the JSON file approach, I figured if creating a new file anyway, might as well be a python module 😛

Are you okay with this change? I also have an alternative implementation if you prefer, where the groups/args are defined in a dict and the arguments are set up with

arg_groups: Dict[str, List[Dict[str, Any]]] = {
    'Options': [
        {
            'flags': ['--http-headers'],
            'nargs': '+',
            'metavar': 'HTTP_HEADER',
            'help': 'Set headers in HTTP requests to the remote host. (example: "Authorization: Basic dXNlcjpwYXNz")',
        },
    ],
    'Typing customization': [
        {
            'flags': ['--base-class'],
            'help': 'Base Class (default: pydantic.BaseModel)',
            'type': str,
        },
        # ...
    ],
    # ...
}

for group_name, args in arg_groups.items():
    group = arg_parser.add_argument_group(group_name)

    for arg in args:
        flags, kwargs = arg.pop('flags'), arg
        group.add_argument(*flags, **kwargs)

[koxudaxi]

@howardj99
I'm sorry I forgot I used python logic when defining the command options.

I agree with the approach.

arg_groups: Dict[str, List[Dict[str, Any]]]

But, Can we use pydantic model or dataclasses.dataclass or TypedDict or NamedTuple for the option definition?

[howardj99]

@howardj99 I'm sorry I forgot I used python logic when defining the command options.

I agree with the approach.

arg_groups: Dict[str, List[Dict[str, Any]]]

But, Can we use pydantic model or dataclasses.dataclass or TypedDict or NamedTuple for the option definition?

You're saying you'd rather go with that approach instead? Right now it's just a bunch of static calls to add_argument() like it was previously. I'm happy to update to the alternative approach but with modeled data instead

[koxudaxi]

@howardj99

You're saying you'd rather go with that approach instead?

I would like to check runtime for typechecking or unittest to make sure each type and the key is correct, even if it is exaggerated.

[howardj99]

@koxudaxi please take a look at this commit and let me know if you prefer it the new way or the old way

[howardj99]

@koxudaxi please take a look at this commit and let me know if you prefer it the new way or the old way

Looks like that broke everything. Weird that it was passing unit tests locally. Would you be ok with the previous implementation, without the dict or JSON data?

[koxudaxi]

I checked your changes.
The __init__ was broken.

pydantic/main.py:526: in pydantic.main.BaseModel.parse_obj
    ???
E   TypeError: ArgumentGroupsV1.__init__() missing 1 required positional argument: 'data'

We will probably get the chance to improve the part of the argument next time.

Thank you very much!!