runopts: infer from TypedDict

[d4l3k]

This adds a new runopts.from_typed_dict method and uses it to generate the runopts from the typed dict field, annotations, default parameters and docstring.

This simplifies adding new fields to schedulers since you only need to define it in a single spot.

I've migrated DockerScheduler to use the new runopts and will do the rest in a follow up PR.

Test plan:

pytest torchx/specs/test/api_test.py torchx/schedulers/test/api_test.py

(torchx) rice@ariel ~/D/torchx (runopts_typedict)> torchx runopts local_docker
local_docker:
    usage:
        [copy_env=COPY_ENV],[image_repo=IMAGE_REPO]

    optional arguments:
        copy_env=COPY_ENV (typing.List[str], None)
            list of glob patterns of environment variables to copy if not set in AppDef. Ex: FOO_*",
        image_repo=IMAGE_REPO (str, None)
            (remote jobs) the image repository to use when pushing patched images, must have push access. Ex: example.com/your/con
tainer

[d4l3k]

cc @Sara-KS for the mcad changes -- should be a no-op, just moving image_repo opt to pull from the DockerWorkspaceMixin

[codecov]

Codecov Report

Merging #708 (fc01593) into main (3267fa9) will decrease coverage by 0.01%.
The diff coverage is 94.11%.

@@            Coverage Diff             @@
##             main     #708      +/-   ##
==========================================
- Coverage   92.54%   92.54%   -0.01%     
==========================================
  Files          86       86              
  Lines        5742     5767      +25     
==========================================
+ Hits         5314     5337      +23     
- Misses        428      430       +2     

Impacted Files	Coverage Δ
torchx/schedulers/docker_scheduler.py	95.83% <ø> (-0.09%)	⬇️
torchx/specs/api.py	97.93% <92.30%> (-0.48%)	⬇️
torchx/schedulers/api.py	94.64% <100.00%> (+0.30%)	⬆️
torchx/schedulers/gcp_batch_scheduler.py	91.62% <100.00%> (ø)
torchx/schedulers/kubernetes_mcad_scheduler.py	94.56% <100.00%> (-0.02%)	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[kiukchung]

LGTM overall, commented on a philosophical question around the new typing.Required and typing.NotRequired types in python 3.11 and their co-usages with Optional

[kiukchung]

Should we use google-style docstring for these like everywhere else in torchx?
"""
Attributes:
copy_env: list of glob patterns of environment variables to copy if not set in AppDef. Ex: FOO_*"
"""

I validated that the runopts.from_typed_dict() parses google-style as well as numpy style docstrings equally as well.

[kiukchung]

this is going to get interesting in python 3.11 with PEP-0655 that introduces typing.Required and typing.NotRequired.

A couple of interesting questions:

1. how do we interpret a Required[Optional[int]]? (disallow it? allow "required" Optional cfgs? or implicitly make this required?)
2. are there differences between declaring a cfg as Optional[int] versus NotRequired[int]? (technically the former would be in the runopt as None by default, and the later would simply not exist in runopt <- this concept doesn't exist in runopt yet so we'd have to default it to None, but then the typing breaks)