Docs on ManualInputConfig and ManualOutputConfig?

[danthegoodman1]

I see examples of using them such as in the README, as well as blog posts like https://www.bytewax.io/blog/chat-gpt-clone-in-python#:~:text=backoff%20pynats%20openai-,NATS%20Input,-To%20start%2C%20let%27s but I haven't found any docs on how you actually use these, and what the classes should look like.

Perhaps I am missing them?

[davidselassie]

In our latest release, v0.16 we have changed Bytewax's API for input and output to better match the common "partitioned input" use case and to have a data model for IO that will eventually allow re-scaling the execution cluster. Unfortunately, we haven't been able to go back and update all of our blog posts with the new APIs yet.

In summary ManualInputConfig and ManualOutputConfig no longer exist and have been most-directly replaced by DynamicInput and DynamicOutput but the latter can not support persisting state across failures. There is a new PartitionedInput and PartitionedOutput that have APIs that help you write correctly functioning connectors over failures which was very hard to do correctly with the Manual versions.

https://www.bytewax.io/docs/reference/upgrading-to-v16#input-changes has a quick overview of what the changes look like and https://www.bytewax.io/blog/custom-input-connector walks through how we built our Kafka input connector on the new API. You can also peruse our other built-in connectors https://github.com/bytewax/bytewax/tree/main/pysrc/bytewax/connectors to get a sense of how to use the API. https://www.bytewax.io/apidocs/bytewax.inputs and https://www.bytewax.io/apidocs/bytewax.outputs have the formal reference material for those APIs.

If you have specific questions on how to use those new APIs, we're happy to help out in the Bytewax community Slack or for you to open another discussion topic with your specific questions! Thanks!

[davidselassie]

Also if you still want to see docs for an older < 0.16 version of Bytewax, there is a version dropdown in the upper left of the docs to let you pick other versions. Versions <= 0.15.1 have the Manual versions of IO. I encourage you to do your work using the latest version, though, as the new APIs are a much better fit for how to think about IO in a dataflow context.

[danthegoodman1]

Great, thank you!