feat: add ray ids to workflows, clean up types

[MasterPtato]

Changes

[graphite-app]

Your org requires the Graphite merge queue for merging into main

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

You can enable merging using labels in your Graphite merge queue settings.

[MasterPtato]

• feat(workflows): add retries and timeouts #860
• feat(workflows): add worker instance failover #854
• feat: run sub workflows in the same process #789
• feat: add compat layer between old ctx and new workflows #788
• feat: add ray ids to workflows, clean up types #787 👈
• fix: cast workflow errors to raw global errors #785
• feat: global error raw variant #784
• feat: add workflows #783
• fix: remove trace from ops #780
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @MasterPtato and the rest of your teammates on Graphite

[AngelOnFira]

Some initial thoughts. These are more "stream of consiousness" than things that nessessarily need to be changed.

[AngelOnFira]

This seems to imply that workflows don't need to be run on the same machine. Are
there any limitations to how this might scale?

Also, it would be cool to see a bit of a written example. Something akin to:

For example, there might be 3 workers, each with their own registry list:

Worker 1:
    - cpu-intensive
    - private-database
Worker 2:
    - private-database
    - public-database
Worker 3:
    - public-database

[MasterPtato]

I'm not sure what those lists are referring to in terms of an example

[AngelOnFira]

This makes it sound like I'd define a registry with several workflows
somewhere, then be able to re-use it in code to put several complex steps into a
single call.

It also sounds like it would be helpful to have a way to easily navigate all
existing registries in a similar way that I might go look for existing ops that do the
thing I want.

[MasterPtato]

then be able to re-use it in code to put several complex steps into a
single call

What do you mean by this?

Registries exist within code and do not need to be searched for anything by the developer. Its a way to tell the worker what workflows you want it to run. You define a registry in the /worker of each pkg and then in the monolith worker you import and merge them together when starting the workflow worker. They dont deal with ops at all

[AngelOnFira]

I'm not sure I understand what "tidiness" means in this context. I'm also
curious about the Activity vs Operation distinction. What can an Activity do
that an operation can't?

Does this basically mean that it's more like separating functions? So an example
of an Activity might be "create a new user", and this uses the operations "find
existing user" and "create new user" and "send user email"?

[NathanFlurry]

biggest thing is: use Box<serde_json::rawvalue> everywhere (essentially a string) instead of serde_json::Value (which does a bunch of heap allocs)

[MasterPtato]

See my comment on raw value

[graphite-app]

Merge activity

• Jun 5, 4:13 PM EDT: MasterPtato added this pull request to the Graphite merge queue.
• Jun 5, 4:26 PM EDT: MasterPtato merged this pull request with the Graphite merge queue.