SAR sparse multiplcation modification due to a breaking change in scipy

[miguelgfierro]

Description

Related Issues

#1954

References

Checklist:

• I have followed the contribution guidelines and code style for this project.
• I have added tests covering my contributions.
• I have updated the documentation accordingly.
• I have signed the commits, e.g. git commit -s -m "your commit message".
• This PR is being made to staging branch AND NOT TO main branch.

[miguelgfierro]

with scipy 1.11.1, and python 3.9 it works:

$ pytest tests/unit/examples/test_notebooks_python.py::test_sar_deep_dive_runs --disable-warnings
================================================= test session starts ==================================================
platform linux -- Python 3.9.16, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/miguel/MS/recommenders
configfile: pyproject.toml
plugins: hypothesis-6.80.0, cov-4.1.0, mock-3.11.1, typeguard-4.0.0, anyio-3.7.0
collected 1 item

tests/unit/examples/test_notebooks_python.py .                                                                   [100%]

============================================ 1 passed, 3 warnings in 7.23s =============================================

With scipy 1.13.0 and python 3.9:

$ pytest tests/unit/examples/test_notebooks_python.py::test_sar_deep_dive_runs --disable-warnings
================================================= test session starts ==================================================
platform linux -- Python 3.9.16, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/miguel/MS/recommenders
configfile: pyproject.toml
plugins: hypothesis-6.80.0, cov-4.1.0, mock-3.11.1, typeguard-4.0.0, anyio-3.7.0
collected 1 item

tests/unit/examples/test_notebooks_python.py .                                                                   [100%]

============================================ 1 passed, 3 warnings in 6.62s =============================================

For benchmarking purpose, using the previous version of scipy 1.10.1, it is slower:

$ pytest tests/unit/examples/test_notebooks_python.py::test_sar_deep_dive_runs --disable-warnings
================================================= test session starts ==================================================
platform linux -- Python 3.9.16, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/miguel/MS/recommenders
configfile: pyproject.toml
plugins: hypothesis-6.80.0, cov-4.1.0, mock-3.11.1, typeguard-4.0.0, anyio-3.7.0
collected 1 item

tests/unit/examples/test_notebooks_python.py .                                                                   [100%]

============================================ 1 passed, 3 warnings in 8.39s =============================================

[miguelgfierro]

Error in python 3.11:

============================= test session starts ==============================
platform linux -- Python 3.11.5, pytest-8.1.1, pluggy-1.4.0
rootdir: /mnt/azureml/cr/j/e62514dcd77647bcb13baecdeaa9a748/exe/wd
configfile: pyproject.toml
plugins: anyio-4.3.0, hypothesis-6.100.0, cov-5.0.0, typeguard-4.2.1, mock-3.14.0
collected 19 items

tests/unit/examples/test_notebooks_python.py s...                        [ 21%]
tests/unit/recommenders/utils/test_notebook_utils.py ..........          [ 73%]
tests/unit/examples/test_notebooks_python.py 

=================================== FAILURES ===================================
__________________________ test_sar_single_node_runs ___________________________

notebooks = ***'als_deep_dive': '/mnt/azureml/cr/j/e62514dcd77647bcb13baecdeaa9a748/exe/wd/examples/02_model_collaborative_filtering...rk_movielens': '/mnt/azureml/cr/j/e62514dcd77647bcb13baecdeaa9a748/exe/wd/examples/06_benchmarks/movielens.ipynb', ...***
output_notebook = 'output.ipynb', kernel_name = 'python3'

    @pytest.mark.notebooks
    def test_sar_single_node_runs(notebooks, output_notebook, kernel_name):
        notebook_path = notebooks["sar_single_node"]
>       execute_notebook(notebook_path, output_notebook, kernel_name=kernel_name)

tests/unit/examples/test_notebooks_python.py:34: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
recommenders/utils/notebook_utils.py:102: in execute_notebook
    executed_notebook, _ = execute_preprocessor.preprocess(
/azureml-envs/azureml_e95b5901212d39f15b2a44eca8fce43f/lib/python3.11/site-packages/nbconvert/preprocessors/execute.py:102: in preprocess
    self.preprocess_cell(cell, resources, index)
/azureml-envs/azureml_e95b5901212d39f15b2a44eca8fce43f/lib/python3.11/site-packages/nbconvert/preprocessors/execute.py:123: in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
/azureml-envs/azureml_e95b5901212d39f15b2a44eca8fce43f/lib/python3.11/site-packages/jupyter_core/utils/__init__.py:165: in wrapped
    return loop.run_until_complete(inner)
/azureml-envs/azureml_e95b5901212d39f15b2a44eca8fce43f/lib/python3.11/asyncio/base_events.py:653: in run_until_complete
    return future.result()
/azureml-envs/azureml_e95b5901212d39f15b2a44eca8fce43f/lib/python3.11/site-packages/nbclient/client.py:1062: in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <nbconvert.preprocessors.execute.ExecutePreprocessor object at 0x149eec95f690>
cell = ***'cell_type': 'code', 'execution_count': 8, 'metadata': ***'execution': ***'iopub.status.busy': '2024-04-08T15:15:06.80546...d_k_items(test, top_k=TOP_K, remove_seen=True)\n\nprint("Took *** seconds for prediction.".format(test_time.interval))'***
cell_index = 16
exec_reply = ***'buffers': [], 'content': ***'ename': 'TypeError', 'engine_info': ***'engine_id': -1, 'engine_uuid': 'bd67981e-ed10-462d-...e, 'engine': 'bd67981e-ed10-462d-bdbd-359a88e1244a', 'started': '2024-04-08T15:15:06.805810Z', 'status': 'error'***, ...***

    async def _check_raise_for_error(
        self, cell: NotebookNode, cell_index: int, exec_reply: dict[str, t.Any] | None
    ) -> None:
        if exec_reply is None:
            return None
    
        exec_reply_content = exec_reply["content"]
        if exec_reply_content["status"] != "error":
            return None
    
        cell_allows_errors = (not self.force_raise_errors) and (
            self.allow_errors
            or exec_reply_content.get("ename") in self.allow_error_names
            or "raises-exception" in cell.metadata.get("tags", [])
        )
        await run_hook(
            self.on_cell_error, cell=cell, cell_index=cell_index, execute_reply=exec_reply
        )
        if not cell_allows_errors:
>           raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E           ------------------
E           with Timer() as test_time:
E               top_k = model.recommend_k_items(test, top_k=TOP_K, remove_seen=True)
E           
E           print("Took *** seconds for prediction.".format(test_time.interval))
E           ------------------
E           
E           ----- stderr -----
E           2024-04-08 15:15:06,821 INFO     Calculating recommendation scores
E           ------------------
E           
E           ---------------------------------------------------------------------------
E           TypeError                                 Traceback (most recent call last)
E           Cell In[8], line 2
E                 1 with Timer() as test_time:
E           ----> 2     top_k = model.recommend_k_items(test,top_k=TOP_K,remove_seen=True)
E                 4 print("Took *** seconds for prediction.".format(test_time.interval))
E           
E           File /mnt/azureml/cr/j/e62514dcd77647bcb13baecdeaa9a748/exe/wd/recommenders/models/sar/sar_singlenode.py:535, in SARSingleNode.recommend_k_items(self, test, top_k, sort_top_k, remove_seen)
E               522 def recommend_k_items(self, test, top_k=10, sort_top_k=True, remove_seen=False):
E               523     """Recommend top K items for all users which are in the test set
E               524 
E               525     Args:
E              (...)
E               532         pandas.DataFrame: top k recommendation items for each user
E               533     """
E           --> 535     test_scores = self.score(test,remove_seen=remove_seen)
E               537     top_items, top_scores = get_top_k_scored_items(
E               538         scores=test_scores, top_k=top_k, sort_top_k=sort_top_k
E               539     )
E               541     df = pd.DataFrame(
E               542         ***
E               543             self.col_user: np.repeat(
E              (...)
E               548         ***
E               549     )
E           
E           File /mnt/azureml/cr/j/e62514dcd77647bcb13baecdeaa9a748/exe/wd/recommenders/models/sar/sar_singlenode.py:357, in SARSingleNode.score(self, test, remove_seen)
E               354 if self.normalize:
E               355     counts = self.unity_user_affinity[user_ids, :].dot(self.item_similarity)
E               356     user_min_scores = (
E           --> 357         np.tile(counts.min(axis=1)[:,np.newaxis], test_scores.shape[1])
E               358         * self.rating_min
E               359     )
E               360     user_max_scores = (
E               361         np.tile(counts.max(axis=1)[:, np.newaxis], test_scores.shape[1])
E               362         * self.rating_max
E               363     )
E               364     test_scores = rescale(
E               365         test_scores,
E               366         self.rating_min,
E              (...)
E               369         user_max_scores,
E               370     )
E           
E           TypeError: 'coo_matrix' object is not subscriptable

/azureml-envs/azureml_e95b5901212d39f15b2a44eca8fce43f/lib/python3.11/site-packages/nbclient/client.py:918: CellExecutionError

[miguelgfierro]

@gramhagen it seems we still have the same error

[gramhagen]

oh, i see. I didn't realize that only occurred on scipy 1.13.
counts should be a csr matrix, but if you get the min it looks like it's converting it back to coo which you can't index with [:, np.newaxis]
at this point the scores are dense arrays, so I think the right solution is to convert the min and max results to arrays as well. I will add a suggestion to fix this

[gramhagen]

replace lines 356-363 of sar_singlenode.py with this (adding toarray())

            user_min_scores = (
                np.tile(counts.min(axis=1).toarray(), test_scores.shape[1])
                * self.rating_min
            )
            user_max_scores = (
                np.tile(counts.max(axis=1).toarray(), test_scores.shape[1])
                * self.rating_max
            )

[miguelgfierro]

We got an error here:

=================================== FAILURES ===================================
_________________________ test_predict[jaccard-False] __________________________

similarity_type = 'jaccard', timedecay_formula = False
train_test_dummy_timestamp = [   UserId  MovieId  Rating   Timestamp
4       2        5     5.0  1535133522
9       2       10     5.0  1535133622
...,    UserId  MovieId  Rating   Timestamp
2       1        3     3.0  1535133482
8       2        9     4.0  1535133602]
header = ***'col_item': 'MovieId', 'col_rating': 'Rating', 'col_timestamp': 'Timestamp', 'col_user': 'UserId'***

    @pytest.mark.parametrize(
        "similarity_type, timedecay_formula", [("jaccard", False), ("lift", True)]
    )
    def test_predict(
        similarity_type, timedecay_formula, train_test_dummy_timestamp, header
    ):
        model = SAR(
            similarity_type=similarity_type, timedecay_formula=timedecay_formula, **header
        )
        trainset, testset = train_test_dummy_timestamp
        model.fit(trainset)
>       preds = model.predict(testset)

tests/unit/recommenders/models/test_sar_singlenode.py:53: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <recommenders.models.sar.sar_singlenode.SARSingleNode object at 0x1509b192c400>
test =    UserId  MovieId  Rating   Timestamp
2       1        3     3.0  1535133482
8       2        9     4.0  1535133602

    def predict(self, test):
        """Output SAR scores for only the users-items pairs which are in the test set
    
        Args:
            test (pandas.DataFrame): DataFrame that contains users and items to test
    
        Returns:
            pandas.DataFrame: DataFrame contains the prediction results
        """
    
        test_scores = self.score(test)
        user_ids = np.asarray(
            list(
                map(
                    lambda user: self.user2index.get(user, np.NaN),
                    test[self.col_user].values,
                )
            )
        )
    
        # create mapping of new items to zeros
        item_ids = np.asarray(
            list(
                map(
                    lambda item: self.item2index.get(item, np.NaN),
                    test[self.col_item].values,
                )
            )
        )
        nans = np.isnan(item_ids)
        if any(nans):
            logger.warning(
                "Items found in test not seen during training, new items will have score of 0"
            )
            test_scores = np.append(test_scores, np.zeros((self.n_users, 1)), axis=1)
            item_ids[nans] = self.n_items
            item_ids = item_ids.astype("int64")
    
        df = pd.DataFrame(
            ***
                self.col_user: test[self.col_user].values,
                self.col_item: test[self.col_item].values,
>               self.col_prediction: test_scores[user_ids, item_ids].getA1(),
            ***
        )
E       AttributeError: 'numpy.ndarray' object has no attribute 'getA1'

recommenders/models/sar/sar_singlenode.py:596: AttributeError

[miguelgfierro]

New error:

$ pytest tests/unit/recommenders/models/test_sar_singlenode.py --disabl
e-warnings
================================================= test session starts ==================================================
platform linux -- Python 3.10.14, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/miguel/MS/recommenders
configfile: pyproject.toml
plugins: hypothesis-6.100.1, anyio-4.3.0, cov-5.0.0, typeguard-4.2.1, mock-3.14.0
collected 36 items                                               FF.FFFFFF.....F.F........                                                                      [100%]

============================================================================== FAILURES ===============================================================================
_______________________________________________________________ test_sar_item_similarity[1-cosine-cos] ________________________________________________________________
threshold = 1, similarity_type = 'cosine', file = 'cos'
demo_usage_data =                  UserId    MovieId     Timestamp  Rating
0      0003000098E85347  DQF-00358  1.433879e+09       1
1   ...4C72  DQF-00248  1.416293e+09       1
11675  00037FFE818E4C72  DAF-00375  1.416293e+09       1

[7189 rows x 4 columns]
sar_settings = {'ATOL': 1e-08, 'FILE_DIR': 'https://recodatasets.z20.web.core.windows.net/sarunittest/', 'TEST_USER_ID': '0003000098E85347'}
header = {'col_item': 'MovieId', 'col_rating': 'Rating', 'col_timestamp': 'Timestamp', 'col_user': 'UserId'}

    @pytest.mark.parametrize(
        "threshold,similarity_type,file",
        [
            (1, "cooccurrence", "count"),
            (1, "cosine", "cos"),
            (1, "inclusion index", "incl"),
            (1, "jaccard", "jac"),
            (1, "lexicographers mutual information", "lex"),
            (1, "lift", "lift"),
            (1, "mutual information", "mi"),
            (3, "cooccurrence", "count"),
            (3, "cosine", "cos"),
            (3, "inclusion index", "incl"),
            (3, "jaccard", "jac"),
            (3, "lexicographers mutual information", "lex"),
            (3, "lift", "lift"),
            (3, "mutual information", "mi"),
        ],
    )
    def test_sar_item_similarity(
        threshold, similarity_type, file, demo_usage_data, sar_settings, header
    ):

        model = SAR(
            similarity_type=similarity_type,
            timedecay_formula=False,
            time_decay_coefficient=30,
            threshold=threshold,
            **header
        )

        # Remove duplicates
        demo_usage_data = demo_usage_data.sort_values(
            header["col_timestamp"], ascending=False
        )
        demo_usage_data = demo_usage_data.drop_duplicates(
            [header["col_user"], header["col_item"]], keep="first"
        )

        model.fit(demo_usage_data)

        true_item_similarity = pd.read_csv(
            sar_settings["FILE_DIR"] + "sim_" + file + str(threshold) + ".csv", index_col=0
        )
        item2index = pd.Series(model.item2index)
        index = item2index[true_item_similarity.index]
        columns = item2index[true_item_similarity.columns]

        if similarity_type == "cooccurrence":
            test_item_similarity = pd.DataFrame(model.item_similarity.todense())
            test_item_similarity = test_item_similarity.reindex(
                index=index, columns=columns
            )
            assert np.array_equal(
                true_item_similarity.astype("float64"),
                test_item_similarity.astype("float64"),
            )
        else:
>           test_item_similarity = pd.DataFrame(model.item_similarity)

tests/unit/recommenders/models/test_sar_singlenode.py:138:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../anaconda/envs/recommenders310/lib/python3.10/site-packages/pandas/core/frame.py:843: in __init__
    data = list(data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <101x101 sparse matrix of type '<class 'numpy.float64'>'
        with 3581 stored elements in COOrdinate format>

    def __iter__(self):
        for r in range(self.shape[0]):
>           yield self[r]
E           TypeError: 'coo_matrix' object is not subscriptable

../../anaconda/envs/recommenders310/lib/python3.10/site-packages/scipy/sparse/_base.py:260: TypeError

[miguelgfierro]

Trying with @anargyri and @SimonYansenZhao to roll back and change the array casting
pytest tests/unit/examples/test_notebooks_python.py::test_sar_single_node_runs

    def score(self, test, remove_seen=False):
        """Score all items for test users.

        Args:
            test (pandas.DataFrame): user to test
            remove_seen (bool): flag to remove items seen in training from recommendation

        Returns:
            numpy.ndarray: Value of interest of all items for the users.
        """

        # get user / item indices from test set
        user_ids = list(
            map(
                lambda user: self.user2index.get(user, np.NaN),
                test[self.col_user].unique(),
            )
        )
        if any(np.isnan(user_ids)):
            raise ValueError("SAR cannot score users that are not in the training set")

        # calculate raw scores with a matrix multiplication
        logger.info("Calculating recommendation scores")
        test_scores = self.user_affinity.toarray()[user_ids, :].dot(
            self.item_similarity
        )

Small example:

user_affinity = csr_array([[1, 2, 0], [0, 0, 3], [4, 0, 5]])
item_similarity = np.array([[1, 1, -1],[1,2,3],[4,5,6]])
user_affinity.dot(item_similarity)

array([[ 3,  5,  5],
       [12, 15, 18],
       [24, 29, 26]])

user_ids = [0, 1]
user_affinity[user_ids, :].dot(item_similarity)

array([[ 3,  5,  5],
       [12, 15, 18]])

[SimonYansenZhao]

@miguelgfierro @anargyri I think numpy or scipy has different behaviors with Python 3.8 and other Python versions, because the same tests in group_notebooks_cpu_001 with Python 3.9+ passed but failed with Python 3.8 after I changed item_similarity from np.array(result) to result.toarray().

And I found that if testing with Python 3.8, the latest supported scipy version is 1.10 which worked with np.array(result), but when testing with Python 3.9, scipy 1.11+ will be installed but works with result.toarray().

[miguelgfierro]

@SimonYansenZhao the world is falling if one can't trust numpy or scipy anymore.

[miguelgfierro]

Awesome! @SimonYansenZhao @anargyri @loomlike can you guys accept the PR? since I started it, I can't accept it