Pytest-friendly Test API #5683
Comments
|
Hi @leycec, Again, thanks for the well thought out post and hopefully we can see this coming to fruition soon! |
willhuang1997
added
type:enhancement
carolinedlu
changed the title
|
Hey @leycec , thanks for opening this issue. You are absolutely right that Streamlit's current public/external-dev support for testing apps is lacking. In fact, it is somewhat lacking for internal use too. As @willhuang1997 mentioned, I have a project in the works for unit testing interactions and more ergonomically querying the state of an app, because writing e2e tests with cypress just to test functionality involving multiple script runs is pretty painful. We only have concrete plans for an initial internal version, but there's a lot of excitement for then making it a public api, and I'm hoping to have it polished and stable enough to make it officially available early next year. It will take a slightly different approach than At a high level, the upcoming API will actually build on the approach This is eliding a lot of detail so this might be hard to answer, but I'm curious if this sounds like it would cover your testing needs, or if there's something missing (so I can account for that when planning future projects). I'd also be happy to ping you once we have an internal implementation in case you want to try it out and give feedback prior to the public release. Just let me know if that interests you. |
|
O frabjous day. Thanks so much for that deep dive into the applause-worthy future of Streamlit QA. Excitement intensifies. That's exactly what we were grepping for:
😏
🍾 🥂 🍻
By Thor, yes. I hadn't actually realized that the Google's
🥳 🎈 🎂
Absolutely. You are speaking of a future utopia that the world needs. That would cover almost everyone else's testing needs, too. StackOverflow is overflowing with questions on this very topic. Okay. Technically, I could only find one. Pragmatically, your proposed API seems to satisfy that question's for automated inspection ala:
The only answer to that question suggests Selenium – probably because that's the only thing that works. Thus, I facepalm.
That sounds great! I'm obsessed with QA in Python and would ❤️ to gently prod Streamlit forward there. |
|
Hi @AnOctopus any news about the unit testing project? thanks |
Streamlit is notoriously hard to py-test at the moment. See, e.g., <streamlit/streamlit#5683>
Streamlit is phenomenal. Everyone knows this. This is why my beautiful science wife 😻 and I are building out our next open-source multiphysics bioelectricity simulator in Streamlit.
Streamlit: yup, it even does bioelectricity.
Streamlit Testing: A Chink in the Armour
But all is not quite so phenomenal on the testing front. Sadly, Streamlit currently offers no official means of testing Streamlit apps from a
pytesttest suite. Technically, there does exist:streamlit-mockpackage. Of course, this package is effectively unmaintained. While we applaud the one lone GitHub user brave enough to starstreamlit-mock, we are not that user.streamlit-mock: Visions of a Better Future
Nobody should use
streamlit-mock, because nobody should use unmaintained things. But everybody should be inspired bystreamlit-mockto create something similar that actually works and is well-maintained, because the core conceits behindstreamlit-mockare sound. To quote theirREADME.md:Goals seem reasonable. Non-goals seem reasonable, too. We applaud these reasonable things.
What Do You Want Us to Do About Your Problems, Bro?
Okay. Let's get down to brass data science tacks. It'd be phenomenal if Streamlit itself offered an official
pytest-friendly test API – lightly inspired bystreamlit_mockbut actually well-maintained and working. An officialgithub.com/streamlit/pytest-streamlitplugin trivially enabling Streamlit apps to be tested underpytestwould be especially phenomenal.pytest-streamlitwould hypothetically provide support – presumably in the form of one or morepytestfixtures – for programmatically running any arbitrary Streamlit app as an integration test under a Streamlit mock API.Indeed, we see that Streamlit itself tests itself with a
pytesttest suite. This sorta suggests that the requisite functionality for testing downstream Streamlit apps withpytestalready exists... within Streamlit itself! Shock twist is shocking. 😮Indeed, we see that the existing
streamlit/lib/tests/streamlit/runtime/scriptrunner/script_runner_test.pyintegration test appears to already be doing most – maybe even all – of the heavy lifting we'd expect from an officialgithub.com/streamlit/pytest-streamlitplugin.Copy-and-pasting the deeply nested functionality of
script_runner_test.pyinto our ownpytesttest suite justifiably frightens us, however. If something resembling that could be published as a stand-alone API for reuse by the general public, we will happily dance with our Maine Coon cats on the frozen lake outside our cottage in a TikTok video generally regarded as deplorable.What We Used to Do No Longer Works, Which Is Sad
Previously, we used to exercise our entire Streamlit app with a trivial integration test in our
pytesttest suite resembling:Tragically, that test now fails with a non-human-readable CPython error resembling:
Needless to say, we no longer run that test.
Thanks for all the escalating UIX brilliance, Streamlit devs! You make the data science world go round. 🌎
Community voting on feature requests enables the Streamlit team to understand which features are most important to our users.
If you'd like the Streamlit team to prioritize this feature request, please use the 👍 (thumbs up emoji) reaction in response to the initial post.
The text was updated successfully, but these errors were encountered: