Evaluate usefulness of pep-593 for structured configs

[omry]

"This PEP introduces a mechanism to extend the type annotations from PEP 484 with arbitrary metadata."

• pep-593
• PEP master Issue with some examples

Some ideas below. Note that this is all pseudo code for now.

Documentation annotation

Doc : Annotated
num: Doc[int, "A number of things"] = 8

Range annotation for valid input range.

NumRange(T, min, max) = Annotated[T, min, max]
StrRange(Choices) = Annotated[str, Choices]
# 0 <= num < 10
num : NumRange(int, 0, 10)

# 0.1 <= num < 0.2
num : NumRange(float, 0.1, 0.2)

# string must be one of "a", "b" or "c".
# Similar to current Enum support.
string : StrRange["a", "b", "c"] = "a"

Size unit annotation

Number can be converted from some common base unit or string representation to the specified unit.

Size[T] = Annotated[int, T]
# Static type error from string to int, but can work around with Any somehow
# Would convert "1kb" to 1024 (bytes as int).
bytes: Size[B] = "1kb"

# Would convert "1kb" to 1
bytes: Size[KB] = "1kb"

# Would convert to 0.001MB or to 0MB, depending on int or float. not sure.
bytes: Size[MB] = "1kb"

File annotations

Not sure about it, File (Or pathlib.Path) should be a properly supported primitive first. but:

must_be_a_file : File[str, IsFile]
must_be_a_dir : File[str, IsDir]
must_exist : File[str, Exist]

[omry]

@till-varoquaux, would love to chat with you about this to see what is currently possible.
I would like to start with the Doc annotation, and I saw multiple suggestion on the the relevant issue, but as far as I can tell many are not actually implemented and the few that are not really satisfactory.

[Jasha10]

I wanted to mention that the field function defined in the dataclasses module has a metadata keyword. The keyword takes a user-defined mapping as input; it is provided, provided "as a third-party extension mechanism".

@dataclass
class MySchema:
    num: int = field(metadata={"doc": "number of trials", "range": [0, 10]})

Using the PEP 593 annotations feels "cooler" than defining metadata via the field function, but I wanted to mention using field as an alternative. This could be more flexible than using Annotated: while Annotated allows the user to attach a sequence of data to a dataclass attribute, dataclasses.field allows the user to attach a mapping of data to a dataclass attribute.

Attr classes support this too (not just dataclasses).

[adriangb]

Not sure what the status is of this, and I haven't ever used omegaconf myself, but I found this issue from other issues and thought it might be relevant to mention https://github.com/annotated-types/annotated-types since it actually implements some of the proposals here and is currently being used by Pydantic.

[Jasha10]

Thanks @adriangb.