hash_funcs for st.cache_data and st.cache_resource #6502

kajarenc · 2023-04-17T18:19:17Z

📚 Context

hash_funcs parameter existed previously in st.cache decorator, and we had multiple requests to return it to new cache_primitives: #6295 , #5939 , #6290

In this PR we return back hash_funcs functionality mostly using behaviour from st.cache.

What kind of change does this PR introduce?
- Bugfix
- Feature
- Refactoring
- Other, please describe:

🧠 Description of Changes

Add bullet points summarizing your changes here
- This is a breaking API change
- This is a visible (user-facing) change

Revised:

Insert screenshot of your updated UI/code here

Current:

Insert screenshot of existing UI/code here

🧪 Testing Done

Screenshots included
Added/Updated unit tests
Added/Updated e2e tests

🌐 References

Does this depend on other work, documents, or tickets?

Issue: Address Support custom hashing functions for st.cache_data and st.cache_resource #6295 (and issues mentioned in that issue)

Contribution License Agreement

By submitting this pull request you agree that all contributions to this project are made under the Apache 2.0 license.

…e hash_funcs

lib/streamlit/runtime/caching/hashing.py

…mitives

sfc-gh-jcarroll · 2023-05-26T20:54:11Z

This looks great from product perspective. I noted a couple of questions / possible concerns here but they are fairly minor:

https://www.notion.so/snowflake-corp/Tech-Spec-for-hash_funcs-st-cache_data-st-cache_resource-939e6e77613743989f0dd5aea92134b7?pvs=4#13eb911cd6a1454cbc910372dc158d28

sfc-gh-jcarroll · 2023-05-26T20:54:42Z

lib/streamlit/runtime/caching/hashing.py

+If you think this is actually a Streamlit bug, please [file a bug report here.]
+(https://github.com/streamlit/streamlit/issues/new/choose)


This doesn't render as a link, it just shows the raw markdown. Not sure if it's a limitation in st.exception or because of the line break.

Good catch, thank you @sfc-gh-jcarroll !
Removing line break works :)

sfc-gh-jcarroll · 2023-06-01T19:56:23Z

Only last question from me is whether there's any low lift thing to make this easier to use for cached methods in a class.

class MyCustomClass:
    def __init__(self, initial_score: int):
        self.my_score = initial_score

    @st.cache_data(hash_funcs={"__main__.MyCustomClass": id})
    def multiply_score(self, x: int) -> int:
        return self.my_score * x

It would be nice at least if "MyCustomClass" string worked without needing __main__ (and same if the class is defined in an imported file rather than the running one, not sure if that causes further weirdness)

This was a common pain point in many of the forum threads (not sure how it was solved before with st.cache), so if it's difficult to discover / use I think we will continue to get feedback about it.

Otherwise the behavior all looks good from product side based on my testing of the whl file ✅

kajarenc · 2023-06-07T13:51:50Z

lib/tests/streamlit/runtime/caching/cache_resource_api_test.py

+            my_obj = MyObj()
+            user_hash_error_func(my_obj)
+
+        expected_message = """unsupported operand type(s) for +=: 'MyObj' and 'int'


I wrote the same tests cases for st.cache_data and st.cache_resource and did not use parameterized common cache tests because expected_message are very verbose and different, so it would look ugly if parametrized.

kajarenc · 2023-06-07T14:25:14Z

@sfc-gh-jcarroll related to method decoration and "__main__.MyCustomClass" syntax, this happen because class name doesn't exist during method definition, so we have to use string instead of just MyCustomClass

For example in the case of a function that uses an argument of type MyCustomClass and defined after it, we could refer to the class name, and not to string.

For module names starting with "__main__", it is normal practice in Python for top-level executing module, and if the class is defined in an imported file rather than the running one, this should not cause weirdness, for example this work correctly:

app.py

import streamlit as st
import hello_world


x = hello_world.MyHelloWorld()

st.write(x.my_method())
st.write(x.my_method())

and hello_world.py

import streamlit as st


class MyHelloWorld:
    @st.cache_data(hash_funcs={"hello_world.MyHelloWorld": id})
    def my_method(self):
        print("IN METHOD")
        return 45

sfc-gh-jcarroll · 2023-06-07T17:59:38Z

Sounds good. That works for this initial update and we can decide later whether we need to improve it. Thank you!

tconkling

This is great!

tconkling · 2023-06-07T22:19:04Z

lib/streamlit/runtime/caching/hashing.py

+    def pretty_print(self):
+        def to_str(v):
+            try:
+                return "Object of type %s: %s" % (type_util.get_fqn_type(v), str(v))


Nit: can we use an f-string here, instead of %?

tconkling · 2023-06-07T22:19:39Z

lib/streamlit/runtime/caching/hashing.py

@@ -84,6 +180,15 @@ def pop(self):
    def __contains__(self, val: Any):
        return id(val) in self._stack

+    def pretty_print(self):


Needs return type annotation

tconkling · 2023-06-07T22:22:59Z

lib/tests/streamlit/runtime/caching/hashing_test.py

+    @parameterized.expand(
+        [("cache_data", cache_data), ("cache_resource", cache_resource)]
+    )
+    def test_recursive_hash_func(self, _, cache_decorator):


Can you add a short docstring explaining what this test is doing?

tconkling · 2023-06-07T22:23:52Z

lib/tests/streamlit/runtime/caching/hashing_test.py

@@ -391,6 +413,72 @@ def test_generator_not_hashable(self):
        with self.assertRaises(UnhashableTypeError):
            get_hash((x for x in range(1)))

+    def test_hash_funcs_acceptable_keys(self):


This could also use a docstring, because the purpose of the test isn't very clear from its name. (Maybe the comment in the middle of the function is sufficient?)

* develop: Bump dependencies of component-lib (streamlit#6830) Make `streamlit config show` honor newlines (streamlit#6758) Update comments on magic.py (streamlit#6833) Add tests for component-lib (streamlit#6580) Fix button height (streamlit#6738) Add `to_pandas` method to convert inputs to pandas DataFrames (streamlit#6668) hash_funcs for st.cache_data and st.cache_resource (streamlit#6502)

hash_funcs parameter existed previously in st.cache decorator, in this PR we bring back hash_funcs functionality to new cache primitives st.cache_data and st.cache_resource mostly using behavior from st.cache.

kajarenc added 2 commits April 17, 2023 22:17

hash_funcs prototype

9dc2352

add UserHashError machinery to @st.cache_data and @st.cache_resourc…

a491314

…e hash_funcs

kajarenc changed the title ~~hash_funcs prototype~~ [WIP] hash_funcs prototype Apr 28, 2023

github-advanced-security bot found potential problems May 4, 2023

View reviewed changes

lib/streamlit/runtime/caching/hashing.py Fixed Show fixed Hide fixed

kajarenc added 5 commits May 5, 2023 19:29

add comment

d5c3ad1

Merge branch 'remote-develop' into hash-funcs

f37b094

Remove legacy details from hash_func implementation for new cache pri…

85c31e2

…mitives

Merge branch 'remote-develop' into hash-funcs

c2a09de

Merge branch 'remote-develop' into hash-funcs

03eaf88

sfc-gh-jcarroll reviewed May 26, 2023

View reviewed changes

kajarenc added 4 commits May 31, 2023 17:25

Merge branch 'remote-develop' into hash-funcs

b8d1d4e

fix broken file a bug report markdown link

adcc3bd

fix broken file a bug report markdown link in old st.cache too

b6ca511

Add tests for hash_funcs

d1ce8b4

kajarenc added 3 commits June 6, 2023 17:32

Merge branch 'remote-develop' into hash-funcs

adc0e6b

add tests for hash_funcs

6024d28

add tests for hash_funcs cache_resource

f433d85

sfc-gh-kjavadyan changed the title ~~[WIP] hash_funcs prototype~~ hash_funcs for st.cache_data and st.cache_resource Jun 6, 2023

fix typo

2cb941e

kajarenc added change:feature impact:users security-assessment-completed labels Jun 6, 2023

remove unused blank line

5696a8c

kajarenc commented Jun 7, 2023

View reviewed changes

clean up

10167b1

kajarenc marked this pull request as ready for review June 7, 2023 14:34

kajarenc requested a review from tconkling June 7, 2023 14:34

tconkling approved these changes Jun 7, 2023

View reviewed changes

fixes after review

ac31562

kajarenc merged commit cd38181 into develop Jun 8, 2023
45 checks passed

kajarenc deleted the hash-funcs branch June 8, 2023 14:22

This was referenced Jun 15, 2023

New caching methods cannot detect the change of df column name #6236

Closed

When using the unique method of pandas, the st.cache_data method is invalid in some cases. #6182

Open

inigohidalgo mentioned this pull request Aug 11, 2023

Update st.cache migrating guide to remove lack of hash_funcs streamlit/docs#779

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hash_funcs for st.cache_data and st.cache_resource #6502

hash_funcs for st.cache_data and st.cache_resource #6502

kajarenc commented Apr 17, 2023 •

edited

sfc-gh-jcarroll commented May 26, 2023

sfc-gh-jcarroll May 26, 2023

kajarenc May 31, 2023

sfc-gh-jcarroll commented Jun 1, 2023

kajarenc Jun 7, 2023

kajarenc commented Jun 7, 2023

sfc-gh-jcarroll commented Jun 7, 2023

tconkling left a comment

tconkling Jun 7, 2023

tconkling Jun 7, 2023

tconkling Jun 7, 2023

tconkling Jun 7, 2023

		If you think this is actually a Streamlit bug, please [file a bug report here.]
		(https://github.com/streamlit/streamlit/issues/new/choose)

hash_funcs for st.cache_data and st.cache_resource #6502

hash_funcs for st.cache_data and st.cache_resource #6502

Conversation

kajarenc commented Apr 17, 2023 • edited

📚 Context

🧠 Description of Changes

🧪 Testing Done

🌐 References

sfc-gh-jcarroll commented May 26, 2023

sfc-gh-jcarroll May 26, 2023

Choose a reason for hiding this comment

kajarenc May 31, 2023

Choose a reason for hiding this comment

sfc-gh-jcarroll commented Jun 1, 2023

kajarenc Jun 7, 2023

Choose a reason for hiding this comment

kajarenc commented Jun 7, 2023

sfc-gh-jcarroll commented Jun 7, 2023

tconkling left a comment

Choose a reason for hiding this comment

tconkling Jun 7, 2023

Choose a reason for hiding this comment

tconkling Jun 7, 2023

Choose a reason for hiding this comment

tconkling Jun 7, 2023

Choose a reason for hiding this comment

tconkling Jun 7, 2023

Choose a reason for hiding this comment

kajarenc commented Apr 17, 2023 •

edited