deliver a better devX

[charlesfrye]

the developer experience is just OK. let's make it awesome.

note: some of these are more suggestion than requirement

Everywhere

• add some minimal tests
• formalize into a devcontainer so we can run on codespaces
  • configure linting better
  • improve Python environment setup -- pyenv virtualenv seems to be on the out, see VS Code docs
• modularize the Makefile into reusable scripts
• modularize the env? as it gets more complicated, will become necessary

Backend

Modal

• use modal serve in development so that the backend endpoint isn't polluted -- also gets faster updates
  • consider making it "mandatory", ie set by the ENV=dev flag
• update the debug command to use modal shell
• change secret name for dev vs prod
• allow local configuration to separate dev and prod, make the MongoDB default database and collection configurable, add a dev version #41
• use Modal environments to separate dev and prod #73

MongoDB

• better handling of dev/prod database separation, make the MongoDB default database and collection configurable, add a dev version #41
• better setup for ad hoc querying add a function in docstore for general query execution #42
• add QoL tooling for ad hoc work with JSON, e.g. jq #54
• write a notebook for expectation testing/data doc generation for the docstore
  • important: connect to vector index so that vector index exploration feeds nicely into it

FastAPI

• Make sure we have a good Swagger UI
  • linking docs across multiple "apps", e.g. the / mount and the /gradio mount
• Look into QoL tools for FastAPI -- Postman? OpenAPI tooling?
• Review other best practices for FastAPI design, focused on DevX
• Improve the Gradio interface

LLM and Vector Index

Testing

• Add a MockLLM that fits LangChain interface? abstract LLM choice, make it configurable #48
  • reduces costs for unit tests and non-e2e integration tests

Evaluation

• Try out AutoEvaluator
• Make at least one simple LLM evaluation test, e.g. "No relevant sources found" or questions about capabilities
• Automate ingress of emoji feedback, automate ingestion of emoji feedback #47

Tooling

• Vector index exploration in W&B, VectorDB to wanDB #5
• LangChain tracing for debugging

Frontend

• Testing of Discord bot -- health check, maybe a query
• make the /health a "hidden command" so we can use it in prod
  - look into interactions library for Discord to see if it's cleaner, especially for streaming responses
• run the Discord bot in Modal #49