[FEATURE] Support transient flows

[georgeh0]

CocoIndex treats transformation as persistent / keep-going tasks that keep maintaining and evolving the outcome on source data / flow change.

Sometimes users want the other way: treat data transformations as transient tasks (fire and forget).

Specifically:

• Add a new argument transient to the @cocoindex.flow_def decorator, e.g.

  @cocoindex.flow_def(name="MyFlow", transient=True)
  def my_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
    ...

• Live update is not supported for a transient flow

• In this mode, both setup and update are transient tasks (fire and forget) without keeping any states in CocoIndex's internal storage.

---

❤️ Contributors, please refer to 📙Contributing Guide.
Unless the PR can be sent immediately (e.g. just a few lines of code), we recommend you to leave a comment on the issue like I'm working on it or Can I work on this issue? to avoid duplicating work. Our Discord server is always open and friendly.

[shresthashim]

hey @georgeh0 , can you assign me this issue?

[shresthashim]

hey @georgeh0 and @badmonster0 , I have raised a PR for this issue. Please review the PR and have it merged please

[georgeh0]

Hi @shresthashim,

Thanks for creating the PR! The functionality achieved there is different from what we expect for this feature request - sorry it's my fault and I think I probably should document the requirement more clearly and in more details to avoid confusion.

Transient flows requested by this feature should support both setup (i.e. create the target) and update (i.e. after processing data from the source, write to the target) operations.
It's called transient as the flow itself is gone after processing (fire and forget). It doesn't track any history, i.e. it doesn't handle any changes - assume source + processing logic are at a snapshot and process it only once and forget.

To implement it, multiple CocoIndex engine components (in Rust) need to be touched.

Please let me know if you still want to work on this. If so, I can share more details about potential ways to implement it. Thanks!

[badmonster0]

Thanks @georgeh0 I'll take it out from hacktoberfest due to the complexity of this issue, and may take long time to resolve. @shresthashim let us know your preference, thanks

[shresthashim]

@georgeh0 @badmonster0 ok :)

[vipulaSD]

Hi @georgeh0 & @badmonster0 shall I work on this ticket?

[badmonster0]

hi @vipulaSD thank you so much for contributing to the issue! We are in the process of upgrading CocoIndex to v1 where it will be more Python native. I'd love to connect and share the design doc and keep you posted on where it is going towards. lmk :)

Once it is out, the transient flow will be out of box! :) so i'll close this for now.

[vipulaSD]

hi @vipulaSD thank you so much for contributing to the issue! We are in the process of upgrading CocoIndex to v1 where it will be more Python native. I'd love to connect and share the design doc and keep you posted on where it is going towards. lmk :)

Once it is out, the transient flow will be out of box! :) so i'll close this for now.

For sure @badmonster0 !. I'm looking forward to see the design docs. Let me ping you through discord