RTC stress tests

[echarles]

I have shared with @hbcarlos the work we are doing with Anaconda to implement stress tests for the RTC features. The work is being done for now in https://github.com/datalayer/jupyter-rtc-test and activity is planned in the coming weeks to reach a level of usability soon.

I have shared with Carlos during JupyterCon that we think that more confirmation is needed before stating it is usable in a more "production" environment. This confirmation can not come from only the existing unit/integration tests, nor from using it in dev calls. We need a real stress test (also in distributed mode with e.g. simulating latency in the network...) of the different layers and components before giving the green light to any users that Jupyter RTC is ready to be used. In terms of priority, we think that this stress tests are needed before adding any new feature to RTC. It sounded to me that Carlos was aligned with that.

We'd love to have anyone join this effort and happy to contribute back when it is ready.

cc/ @jupyterlab/jupyterlab-council and @fperez

[fcollonval]

Any tests are good tests. Thanks for investing resources to improve this.

About the green light, I'm afraid we have sent contradictory messages about the feat. readiness. But I don't have strong objection to for example keep the jupyter-collaboration package in pre release a bit longer giving time for these tests to be a thing.

I have one organizational reservation for the tests project location. It would better to place it in one of GitHub Jupyter organizations.

[afshin]

I think it's okay if the jupyter-collaboration package takes a little longer to reach 1.0 final because it won't slow down JupyterLab 4.0 final.

At JupyterCon, when @fperez asked (publicly, in the JupyterLab talk) about the reliability and readiness of RTC features, I tried to convey two ideas:

• these features are safe
• they may be rough around the edges, still

If folks at Anaconda have been doing work to create more realistic test loads for RTC, I'd echo @fcollonval's suggestion to upstream this work and also add that Jupyter would be grateful to have it bolster the reliability of RTC features before full-blown release. Thanks @echarles!

[echarles]

If folks at Anaconda have been doing work to create more realistic test loads for RTC, I'd echo @fcollonval's suggestion to upstream this wor

Thx @afshin and @fcollonval for your feedbacks. Those stress tests are still for now WIP and not yet ready to be reviewed, used nor drawing any conclusion. I opened this issue for more visibility and awareness of this initiative and also based on the conversation I had with Carlos. I will post more here as soon as this can be reviewed/discussed in more details.

[hbcarlos]

@echarles Thank you for sharing your work on RTC. Indeed, we should focus on the efforts on testing and caching bugs.

The current implementation of RTC is safe and usable. Of course, there are a few bugs in corner cases. Nevertheless, we should release it as soon as possible. We need people using it and discovering bugs. One of the reasons for moving RTC into an extension was to be able to iterate quickly. My idea is to release jupyter_collaboration v1 as soon as JupyterLab v4 is out.

Feel free to open a PR in https://github.com/jupyterlab/jupyter_collaboration with your work. Once it is ready, I will review it.

Thank you again for working on this. We need more people looking into RTC.

[jtpio]

My idea is to release jupyter_collaboration v1 as soon as JupyterLab v4 is out.

This sounds good 👍 And as new issues are being found and fixed there can be as many 1.0.x releases as needed.

[afshin]

If we can give guidance about the order of magnitude of users you can expect to support it would be helpful. This shouldn't block 1.0. Having capacity advice in our docs very soon would be good for everybody.

[echarles]

Just. read minutes meeting of yesterday JupyterLab call with @fperez highlighting a few more RTC issues.

I have been progressing well on the RTC stress test adding a UI on top of the unit and integration tests that simulate real distributed load which is not achieved today in the various RTC repositories.

There is a lot to discuss and share, but my overall feeling based on my tests is that the current state of Jupyter RTC is not yet ready for production. Part of the goal of that initiative is to clearly identify failure cases and bring fixes.

I was planning to join next week JupyterLab call to show the UI and share the discoveries, but I am not sure this is the best place to discuss.

[hbcarlos]

Thanks a lot, @echarles. The screencast looks great!

Please feel free to open a PR to jupyter-collaboration once it is ready, and I'll take a look at it.

[ellisonbg]

Wow Eric, amazing work, looking forward to learning more!

…

On Thu, Jun 29, 2023 at 2:51 AM Carlos Herrero ***@***.***> wrote: Thanks a lot, @echarles <https://github.com/echarles>. The screencast looks great! Please feel free to open a PR to jupyter-collaboration once it is ready, and I'll take a look at it. — Reply to this email directly, view it on GitHub <#14532 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAGXUCT2G6XKJ4KVYM7RETXNVF3LANCNFSM6AAAAAAYBBSRBY> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

-- Brian E. Granger Senior Principal Technologist, AWS AI/ML ***@***.***) On Leave - Professor of Physics and Data Science, Cal Poly @ellisonbg on GitHub

[jtpio]

cc @davidbrochart who might also be interested in these stress tests

[davidbrochart]

@echarles I saw you created this repo: https://github.com/datalayer/jupyter-rtc-test.

• Is it up-do-date with the current jupyter-collaboration? I see you still use ypy, but we switched to pycrdt.
• Did you find any bug in RTC? I can't find any issue from you.