[job submission] Temporarily make pydantic imports conditional

[jiaodong]

Diff Summary

Current job submission module run on ray default but user might not have pydantic installed. This diff aims to make it lazy import and removed serve import.

Related issue number

closes #19792

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[simon-mo]

Alternatively just add pydantic to ray default?

[jiaodong]

Alternatively just add pydantic to ray default?

i don't see downside of this. where's the precise place i need to add it -- requirements_default.txt?

[jiaodong]

nvm pydantic is surprisingly large https://pypi.org/project/pydantic/#files for how simple the problem it appears to be solving ... so i don't think adding it to ray default is an option

[simon-mo]

Pydantic is large because it uses cython to compile the whole codebase. You should be able to add it to setup.py, look for extra_requires.

I think you should either make it part of ray[default] or add a field set for ray[jobs].

[edoakes]

@richardliaw @ericl thoughts on adding pydantic as a dependency to ray[default]? We would like to use this for schema validation in job submission.

[jiaodong]

so we discussed this during standup and decided to go with lazy import for now but switch to jsonschema soon. We might need pydantic again if we want to migrate dashboard modules to fastapi sometime.

[richardliaw]

how big is the pydantic footprint? feel like it should be ok

…

On Thu, Oct 28, 2021 at 10:02 AM Jiao ***@***.***> wrote: so we discussed this during standup and decided to go with lazy import for now but switch to jsonschema soon. We might need pydantic again if we want to migrate dashboard modules to fastapi sometime. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19827 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCRZZJ2RAXQAVVZOSCWVK3UJF6Z3ANCNFSM5G4AEPZQ> .

[edoakes]

@richardliaw it's like 30MiB...

[edoakes]

Ok, hopefully the last merge conflict here... @jiaodong can you ping me if/when tests are passing?

[pcmoritz]

There is another linting error, @architkulkarni can you fix that? This should be merged ASAP, it is currently blocking the dashboard from working on master :)

[edoakes]

Waiting for CI 😴

[edoakes]

test_client unrealted