Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add realtime/historical adapter example with wikimedia data #225

Merged
merged 3 commits into from
Jun 7, 2024

Conversation

melissawm
Copy link
Contributor

This is a more complete example of csp features, highlighting how to write a realtime adapter and how to use the existing CSVReader adapter for historical data.

I'd be happy to hear if there are other things we want to highlight in this example, or if there are better ways to present the information. This notebook does not include the outputs since they can be very large and include visual elements. I would recommend downloading and running locally.

@melissawm melissawm added the part: documentation Improvements/bugs/changes to documentation label May 9, 2024
Copy link
Collaborator

@AdamGlustein AdamGlustein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently can't run the example in a Python 3.8 environment, please see the below comments.

examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
examples/02_intermediate/wikimedia.ipynb Outdated Show resolved Hide resolved
Copy link
Member

@timkpaine timkpaine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs tests, needs adding to examples/02_intermediate/README.md

Copy link
Member

@timkpaine timkpaine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can initialize your table statically with a schema (you also dont need a Table, you can interact directly with the widget)

{"servername": str, "timestamp": datetime, "event", str, "servername_count": int}

@melissawm
Copy link
Contributor Author

Needs tests, needs adding to examples/02_intermediate/README.md

By tests do you mean something like https://pytest-notebook.readthedocs.io/en/latest/? Do you have another tool in mind?

One thing I have encountered here is that sometimes the realtime example will hang and kill the jupyter kernel. Maybe I am not ending the thread correctly, so if you have thoughts on that it would be great.

Copy link
Member

@timkpaine timkpaine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will discuss testing and org more next week
xref: #246 (comment)

@timkpaine timkpaine added the tag: needs squash PRs that need to be cleaned/squashed label Jun 6, 2024
@melissawm
Copy link
Contributor Author

I have simplified the widget definition and fixed the historical data to be 2 days before execution of the notebooks.

I suggest the tests be addressed in a follow-up so this can be merged as discussed. I can't merge myself because of the requested changes, but feel free to merge any time.

Signed-off-by: Melissa Weber Mendonça <melissawm@gmail.com>

Improve realtime adapter based on feedback from reviewer

Signed-off-by: Melissa Weber Mendonça <melissawm@gmail.com>

Update wikipedia example with a custom PullInputAdapter implementation

Signed-off-by: Melissa Weber Mendonça <melissawm@gmail.com>

Add csp.ticked and remove output from node

Signed-off-by: Melissa Weber Mendonça <melissawm@gmail.com>

Remove counter from example.

Signed-off-by: Melissa Weber Mendonça <melissawm@gmail.com>

Remove pandas from example

Signed-off-by: Melissa Weber Mendonça <melissawm@gmail.com>

Simplify widget definition and fix selection of past data

Signed-off-by: Melissa Weber Mendonça <melissawm@gmail.com>

Move to end-to-end folder

Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
@timkpaine timkpaine dismissed stale reviews from robambalu, AdamGlustein, and themself June 6, 2024 18:07

stale

@timkpaine timkpaine requested a review from ptomecek June 6, 2024 18:29
Signed-off-by: Melissa Weber Mendonça <melissawm@gmail.com>
@timkpaine timkpaine requested a review from robambalu June 7, 2024 14:56
Copy link
Collaborator

@robambalu robambalu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Signed-off-by: Melissa Weber Mendonça <melissawm@gmail.com>
@robambalu robambalu self-requested a review June 7, 2024 15:17
@timkpaine timkpaine merged commit 370ef6b into Point72:main Jun 7, 2024
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
part: documentation Improvements/bugs/changes to documentation tag: needs squash PRs that need to be cleaned/squashed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants