fix #8842: add jetstream/analysis schemas

[mikewilli]

Because

• We want to have a better data contract between Jetstream and Experimenter

This commit

• Creates schemas for the data products that Jetstream produces
• Adds a general python package structure with basic schema tests

[jaredlockhart]

Whoaaaa neeeaaattt it's beautiful 🥹

Okay so we should add some stuff like formatting/linting/tests etc, do you want to do some/any of that in this pr or just get this landed and hook more of that stuff up later?

Also I guess we should setup infrastructure to publish this to pypi and then consume it in experimenter/jetstream? That we can definitely do as followups.

But this is exactly what I was hoping for it looks great 🎉 🎉 🎉

[mikewilli]

Yea, so some of that I was considering for future PRs (publish to pypi, consumption in jetstream/experimenter).

For formatting/linting, I guess I need to ensure that our existing stuff runs against this new dir, and that makes sense to setup here/now.

For tests, do you want to see some tests specific to the schemas themselves, or would it be better to wait until we're using the schemas and then write tests of the functionality they enable alongside that?

[jaredlockhart]

I think a minimal test of just having an example json that stresses the whole schema and running them through it would be nice here. But since this is living at the top level you'll probably need to setup something around it to get pytest going, maybe just a poetry project? Maybe wrap it in a Docker container to ensure there's a stable Python environment for it? (which is what Docker was actually invented for, funnily enough)

[scholtzan]

There are some attributes that looking in Jetstream are set to Optional, so I think they should be optional here too.

Other than that, looks great.

[jaredlockhart]

Tried this on my local and got:

E   ModuleNotFoundError: No module named 'mozilla_nimbus_schemas'

[mikewilli]

Pretty sure I fixed this in the latest commit, but test this out again if you get a chance.

[mikewilli]

Nope, good catch. Thanks!

[b4handjr]

I think the majority of the tests here aren't really needed. Pydantic has great coverage for most of what you are testing. I think the json parts of the tests would be good to keep, as you did write a custom json_loads function.

[mikewilli]

I think the majority of the tests here aren't really needed. Pydantic has great coverage for most of what you are testing. I think the json parts of the tests would be good to keep, as you did write a custom json_loads function.

Yea, makes sense to me. I wrote some of them partly to test how it worked, but you're right! I'll take a sweep and see what I can remove.

[jaredlockhart]

Reviewed again and everything looks great 🎉

So followup are we gonna publish this to pypi? 😱