k_selection_plot and consensus - TypeError: H should have the same dtype as X.

[d-laub]

Hello,

I'm encountering the following error while running k_selection_plot.

I'm working with RNA-Seq data from multiple ~homogeneous tissue samples, rather than scRNA-Seq.

Here are my commands leading up to calling k_selection_plot:

python $CNMF prepare --output-dir $OUTPUT_DIR \
--name $OUTPUT_PREFIX \
-c $COUNTS \
--tpm $TPM \
-k $K_LIST \
--n-iter $N_ITER \
--total-workers $TOTAL_WORKERS \
--seed $SEED \
--densify

# This ran as a SLURM array job
python $CNMF factorize --output-dir $OUTPUT_DIR \
--name $OUTPUT_PREFIX \
--worker-index $WORKER_ID

python $CNMF combine --output-dir $OUTPUT_DIR \
--name $OUTPUT_PREFIX

And here is the traceback for the error after the call:

python $CNMF k_selection_plot --output-dir $OUTPUT_DIR \
--name $OUTPUT_PREFIX

Traceback (most recent call last):
  File "/cellar/users/dlaub/.programs/cnmf/cnmf.py", line 883, in <module>
    cnmf_obj.k_selection_plot()
  File "/cellar/users/dlaub/.programs/cnmf/cnmf.py", line 696, in k_selection_plot
    stats.append(self.consensus(k, skip_density_and_return_after_stats=True).stats)
  File "/cellar/users/dlaub/.programs/cnmf/cnmf.py", line 556, in consensus
    nmf_kwargs=refit_nmf_kwargs)
  File "/cellar/users/dlaub/.programs/cnmf/cnmf.py", line 410, in _nmf
    (usages, spectra, niter) = non_negative_factorization(X, **nmf_kwargs)
  File "/cellar/users/dlaub/anaconda3/envs/py3/lib/python3.7/site-packages/sklearn/utils/validation.py", line 73, in inner_f
    return f(**kwargs)
  File "/cellar/users/dlaub/anaconda3/envs/py3/lib/python3.7/site-packages/sklearn/decomposition/_nmf.py", line 1044, in non_negative_factorization
    "{}.".format(H.dtype))
TypeError: H should have the same dtype as X. Got H.dtype = float64.

Could I get some help with this? Thanks!

[d-laub]

I think the --densify flag might be causing the issue, since when I run try code that loads the counts and tpm matrices into scanpy objects, the dtype of X is float32 (the default for AnnData()).

Further, when I reimport the data from the .h5ad object they are written to in save_norm_counts() their datatype changes to NumPy's '<f8'.

I edited the consensus function within cnmf.py to cast the dtype of norm_counts.X to 'float64' and k_selection_plot ran without errors.

[d-laub]

I had another similar issue calling consensus (weirdly only for some runs and not others). On line 580 in the original cnmf.py I checked if tpm.X.dtype is 'float64' and changed it to 'float64' otherwise and it works now.

[Pentayouth]

In case someone doesn't know
add
norm_counts.X = norm_counts.X.astype(np.float64)
at cnmf.py line 503
will solve this

[jasminelmah]

I did run into this issue for step 4 of the README tutorial, and adding the line suggested by Pentayouth worked. I've now run into what seems to be a related issue, for step 5 of the README. I am doing a test run with the PBMC example data from Seurat, which I first converted to an h5ad file before going through the README.

Command: python3 ./cnmf.py consensus --output-dir ./example_data --name example_cNMF --components 10 --local-density-threshold 0.01 --show-clustering

Traceback:

(base) vpn1722517526:example_PBMC jasmineliannemah$ python3 ./cnmf.py consensus --output-dir ./example_data --name example_cNMF --components 10 --local-density-threshold 0.01 --show-clustering
/Library/Python/3.7/site-packages/sklearn/utils/deprecation.py:143: FutureWarning: The sklearn.decomposition.nmf module is deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.decomposition. Anything that cannot be imported from sklearn.decomposition is now part of the private API.
warnings.warn(message, FutureWarning)
Traceback (most recent call last):
File "./cnmf.py", line 882, in
cnmf_obj.consensus(k, args.local_density_threshold, args.local_neighborhood_size, args.show_clustering)
File "./cnmf.py", line 558, in consensus
nmf_kwargs=refit_nmf_kwargs)
File "./cnmf.py", line 410, in _nmf
(usages, spectra, niter) = non_negative_factorization(X, **nmf_kwargs)
File "/Library/Python/3.7/site-packages/sklearn/utils/validation.py", line 72, in inner_f
return f(**kwargs)
File "/Library/Python/3.7/site-packages/sklearn/decomposition/_nmf.py", line 1044, in non_negative_factorization
"{}.".format(H.dtype))
TypeError: H should have the same dtype as X. Got H.dtype = float64.

It has been a while since I've worked with Python so I'm struggling to understand how to edit the code. Thanks for any help!

[avishai987]

In case someone doesn't know add norm_counts.X = norm_counts.X.astype(np.float64) at cnmf.py line 503 will solve this

Hi, could it be that it's not the right line number? now it seems to raise name error on norm_counts.

P.S: I added it at the beginning of factorize method, and now it works.

[qinting]

Hi, I encountered the same issue using the most recent version of cnmf. I also tried all suggestions listed above, however, the problem still existed. Could anyone please let me know how to solve the problem? Where exactly I should add norm_counts.X = norm_counts.X.astype(np.float64) in cnmf.py?

[ShaoboKang]

Hi，I want to know where exactly are you add norm_counts.X = norm_counts.X.astype(np.float64) in cnmf.py?

[avishai987]

Hi，I want to know where exactly are you add norm_counts.X = norm_counts.X.astype(np.float64) in cnmf.py?

Hi, I encountered the same issue using the most recent version of cnmf. I also tried all suggestions listed above, however, the problem still existed. Could anyone please let me know how to solve the problem? Where exactly I should add norm_counts.X = norm_counts.X.astype(np.float64) in cnmf.py?

Find cnmf.py. At the beginning of the factorize method ( def factorize(self,worker_i=0, total_workers=1,):), one of the parameters is "norm_counts ". Insert "norm_counts.X = norm_counts.X.astype(np.float64)" as the first line of the method, so it will fix norm_counts.

[ShaoboKang]

Okay, thanks for your reply, I'll try it. Hope you happy today. ***@***.*** ***@***.***

…

---- Replied Message ---- From Avishai ***@***.***> Date 04/10/2023 04:16 To ***@***.***> Cc ***@***.***>***@***.***> Subject Re: [dylkot/cNMF] k_selection_plot and consensus - TypeError: H should have the same dtype as X. (#8) Hi，I want to know where exactly are you add norm_counts.X = norm_counts.X.astype(np.float64) in cnmf.py? Hi, I encountered the same issue using the most recent version of cnmf. I also tried all suggestions listed above, however, the problem still existed. Could anyone please let me know how to solve the problem? Where exactly I should add norm_counts.X = norm_counts.X.astype(np.float64) in cnmf.py? Find cnmf.py. At the beginning of the factorize method ( def factorize(self,worker_i=0, total_workers=1,):), one of the parameters is "norm_counts ". Insert "norm_counts.X = norm_counts.X.astype(np.float64)" as the first line of the method, so it will fix norm_counts. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[honghh2018]

Hi everyone,
I get the same error in cnmf 1.3.4
i follow above solution, but it still get error
i add the norm_counts.X = norm_counts.X.astype(np.float64) in cnmf.py below picture

and how to fix this error?
Best
hanhuihong