-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add ray ids to workflows, clean up types #787
Merged
graphite-app
merged 1 commit into
main
from
05-09-feat_add_ray_ids_to_workflows_clean_up_types
Jun 5, 2024
+507
−256
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
# Overview | ||
|
||
Workflows are designed to provide highly durable code executions for distributed systems. The main goal is to allow for writing easy to understand multi-step programs with effective error handling, retryability, and a rigid state. | ||
|
||
## Goals | ||
|
||
**Primary** | ||
|
||
- Performance | ||
- Quick iteration speed | ||
- Architectural simplicity (only depends on CockroachDB) | ||
|
||
**Secondary** | ||
|
||
- Easy to operate, managable via simple SQL queries | ||
- Easier to write, understand, and maintain than event-driven architectures | ||
- Rust-native | ||
- Run in-process and as part of the binary to simplify architecture | ||
- Leverage traits to reduce copies and needless ser/de | ||
- ## Use native serde instead of Protobuf for simplicity (**this comes at the cost of verifiable backwards compatibility with protobuf**) | ||
|
||
## Use cases | ||
|
||
- Billing cron jobs with batch | ||
- Creating servers | ||
- Email loops | ||
- Creating dynamic servers | ||
- Automating Cloudflare APIs (Cloudflare workers, DNS, issuing SSL certs) | ||
|
||
## Relation to existing Chirp primitives | ||
|
||
### Messages | ||
|
||
Workflows replace the use case of messages for durable execution, which is almost all uses of messages. | ||
|
||
The biggest pain point with messages is the lack of a rigid state. Message executions always match the following outline: | ||
|
||
1. Read whatever data is required | ||
2. Perform some action(s) | ||
3. Update data as needed | ||
4. Finish (possibly publish more messages) OR upon failure, start all over at #1 | ||
|
||
The issue with this is that messages do not have any knowledge of messages that came before them, their own previous failed executions, or even other messages of the same system executing in parallel. Without thorough manually written sync checks and consistency validations (which are verbose and hard to follow), this type of execution often results in an overall broken state of whatever system the message is acting on (i.e. matchmaking, server provisioning). | ||
|
||
**Once a broken state is reached, the retry system for messages _practically never_ successfully retries the message.** | ||
|
||
### Cross-package hooks | ||
|
||
We currently use messages for hooking in to events from other workflows so we don't have to bake in support directly. | ||
|
||
This is potentially error prone since it makes control flow more opaque. | ||
|
||
We will use sub workflows instead. | ||
|
||
## Post-workflow message uses | ||
MasterPtato marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Messages should still be used, but much less frequently. They're helpful for: | ||
|
||
- Real-time Data Processing | ||
- Complex Event Processing (CEP) | ||
- Data Transformation and Enrichment | ||
- Continuous Data Integration | ||
- Real-time Monitoring and Alerting | ||
- High-throughput, Low-latency Processing |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to imply that workflows don't need to be run on the same machine. Are
there any limitations to how this might scale?
Also, it would be cool to see a bit of a written example. Something akin to:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what those lists are referring to in terms of an example