Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

demo and test.py: remove dbsp python dependency #1230

Merged
merged 2 commits into from
Jan 19, 2024
Merged

demo and test.py: remove dbsp python dependency #1230

merged 2 commits into from
Jan 19, 2024

Conversation

snkas
Copy link
Contributor

@snkas snkas commented Jan 5, 2024

The goal of this PR is to remove the dbsp python dependency across the demos and test.py.

Largest changes:

  • Removes python dbsp dependency from demos and test.py
  • Main demo README with introduction and overview
  • Adds api_helpers python module for repeated functions
  • Replaces --actions argument with a single --start argument for demos where it is possible
  • Change the use case of test.py for clarity to focus on testing the various API interactions
  • Removes demo shell scripts demo.sh, create_demo_projects.sh, and prepare_demo_data.sh as they are not used elsewhere

Notes:

  • The api_helpers crate can be removed once PUT is implemented. Especially the create_* functions. The compile_program and start_pipeline could be rewritten to a few lines that are duplicated.
  • Once the Python libraries are refactored out, the integration tests should be expanded to cover more API interactions (future PR)

Copy link
Contributor

@ryzhyk ryzhyk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

I think this deserves an entry in CHANGELOG (and release notes): "Create demo pipelines with a single shell command".

@@ -0,0 +1,48 @@
# Demo
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like with this PR, anyone should be able to easily download the Feldera repo and run the demos, so it would help to have even more high-level "Getting started" instructions telling people who are not familiar with our internal architecture how to actually create all required services (I guess it's docker compose or start-manager.sh?) or pointing them to appropriate docs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, we can update the docker compose to have a profile and container entry for each demo , and then standardize how to call them.

It could be already this PR or a next one.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even better: add a very short section on the top-level README, right after the instructions for how to bring up the pipeline manager. E.g.:

### Run a demo

Once you've brought up a pipeline-manager using the above instructions, you can run one of our many demos. For example:

$: <one command for the most simple demo>

This command will bring up a program named foo and a pipeline named bar. You can view them at <urls>.

Once you've tried this, we suggest exploring the full list of our demos [here](link-to-this-folder).

python/api_helpers/README.md Outdated Show resolved Hide resolved
@@ -0,0 +1,48 @@
# Demo
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even better: add a very short section on the top-level README, right after the instructions for how to bring up the pipeline manager. E.g.:

### Run a demo

Once you've brought up a pipeline-manager using the above instructions, you can run one of our many demos. For example:

$: <one command for the most simple demo>

This command will bring up a program named foo and a pipeline named bar. You can view them at <urls>.

Once you've tried this, we suggest exploring the full list of our demos [here](link-to-this-folder).

python/api_helpers/README.md Outdated Show resolved Hide resolved
@lalithsuresh lalithsuresh force-pushed the names-in-urls branch 5 times, most recently from ba92285 to b43eeeb Compare January 16, 2024 22:16
Base automatically changed from names-in-urls to main January 16, 2024 22:23
@snkas
Copy link
Contributor Author

snkas commented Jan 18, 2024

Updated to use PUT requests instead of the api_helpers library (which is now removed).

@snkas snkas force-pushed the demo-rework branch 6 times, most recently from eb1ad5f to ee287c0 Compare January 19, 2024 21:24
- Removes python dbsp dependency from demos and test.py
- Main demo README with introduction and overview
- Replaces `--actions` argument with a single `--start` argument for
  demos where it is possible
- Change the use case of test.py for clarity to focus on testing the
  various API interactions
- Removes demo shell scripts demo.sh, create_demo_projects.sh, and
  prepare_demo_data.sh as they are not used elsewhere

Signed-off-by: Simon Kassing <simon.kassing@feldera.com>
Signed-off-by: Simon Kassing <simon.kassing@feldera.com>
@lalithsuresh
Copy link
Collaborator

LGTM

@snkas snkas merged commit 3290f42 into main Jan 19, 2024
5 checks passed
@snkas snkas deleted the demo-rework branch January 19, 2024 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants