Upgrade schema information retrieval and caching

[rijkvanzanten]

Currently, we introspect the whole schema on the first request, and store the result in the cache. This cached schema overview object is then passed around to all the services so they can rely on the schema information in operation.

This comes with a handful of downsides:

• Introspecting the database schema fresh can take quite a while
  Depending on the database vendor used, it can take anywhere from 30ms to 1.5 seconds to introspect the full database schema. Even though we heavily cache it, that's a non-trivial amount t of time to wait for the schema to be known
• The output schema object is pretty big. Depending on your schema, the overview object can easily be over half a million characters, which in turn causes:
• Caching is very fast for smaller chunks of data with large read/write ops. However, storing a single massive value can easily cause bottlenecks it the connection to/from the caches
• The full schema is read even when it's not needed / only partially used

To improve on this, we should:

• Introspect (and cache) parts of the schema when they're actually requested, rather than all ahead of time
• Store the schema information in smaller chunks (eg per collection / field) in the caches

[rijkvanzanten]

Linear: ENG-205