Should we focus more on sources in documentation?

[sploiselle]

Good technical communication talks to users about what they want to accomplish and guides them along the way––and it starts with whatever they have at the outset of that journey.

Right now, our documentation focuses on sources as an API component––making it a subset of the SQL documentation. However, if you think about the users' experience and mental model, sources are the "thing" they have data in that they want to connect to Materialize.

Part of this is a by-product of Materialize's value proposition: we want to focus on materialized views, because that is what we do that no one else can. On a user's journey, though, the materialization of the views is simply an implementation detail that lets them accomplish some other goal. Another part of this is that when I start drafting documentation, I understood the plan to be exclusively supporting Debezium data over Kafka––i.e. there wasn't much of a story to tell there. As my understanding of the ecosystem has matured, I see that this was insufficient, and the implicit bias this created in the docs provides potential pitfalls for users in getting started with the product.

I believe this ultimately means that the documentation should instead shift its focus on helping users understand connecting their existing sources to Materialize. This also clarifies what the next step ought to be: providing more extensive detail around the kinds of transformations you can accomplish, given your existing data.

Given more work on SQL, this isn't a lift I think I'll be able to accomplish alone, but is something I wanted to put some feelers out for the idea before making sketches.

cc @rjnn @awang

[awang]

Big +1 on this! This leaves off sinks (ie what to do post-Materialize), but it's also scope creep, so I think we should defer sinks until later.

One challenge we will face writing this now is that the specifics of CDC-related sources are still being determined. Ie, there are multiple reasonable approaches, and we're still fleshing many of them out. But I think we have sufficient clarity to put something good out there, and if we keep in mind that some parts may be less-certain we'll be okay.