Remove all usages of numpy.compat ref #13880 #13889

harishsdev · 2021-04-18T14:30:59Z

Reference issue

#13880

What does this implement/fix?

Remove all usages of numpy.compat(for asbytes and asstr functionality)

Additional information

rgommers

Hi @harishsdev, thanks for the PR. This isn't quite right - I explained in more detail what to do.

Also, please write your code in PEP8 compatible fashion (e.g., spaces after commas).

rgommers · 2021-04-18T19:45:12Z

scipy/_lib/_util.py

+def asbytes(s):
+    if isinstance(s, bytes):
+        return s
+    return str(s).encode('latin1')


These are py2-py3 compatibility functions. Given that we only use Python 3, we should not need them. When removing them, the idea is to actually figure out if something is a string or bytes.

I'll give you one example for idl.py.

rgommers · 2021-04-18T19:47:51Z

scipy/io/idl.py

@@ -31,7 +31,7 @@

 import struct
 import numpy as np
-from numpy.compat import asstr
+from scipy._lib._util import asstr


You can remove this import. Then look at where it is used. In this case in one place:

if length > 0: chars = _read_bytes(f, length) _align_32(f) chars = asstr(chars) else: chars = ''

Now, you look at the file mode with which f was opened. In this case that was f = open(file_name, 'rb'). So f.read() is guaranteed to give bytes. Hence you can change the code to:

if length > 0: chars = _read_bytes(f, length).decode('latin1') _align_32(f) else: chars = ''

Then you run the tests to confirm - without .decode('latin1') they all fail, with the decode added they all pass.

harishsdev · 2021-04-19T19:30:53Z

i tried to run tests using following command but i am facing issue,

if its exists,please guide me how to proceed

(pyart_env) PS D:\one\scipy> python .\runtests.py
Building, see build.log...
scipy/linalg/_generate_pyx.py: all files up-to-date
scipy/special/_generate_pyx.py: all files up-to-date
Running scipy\linalg_generate_pyx.py
Running scipy\special_generate_pyx.py
Running scipy\stats_generate_pyx.py
scipy\cluster_hierarchy.pyx has not changed
scipy\io\matlab\mio_utils.pyx has not changed
scipy\io\matlab\streams.pyx has not changed
scipy\linalg\cython_blas.pyx has not changed
scipy\linalg\cython_lapack.pyx has not changed
scipy\interpolate_bspl.pyx has not changed
scipy\linalg_matfuncs_sqrtm_triu.pyx has not changed
scipy\linalg_solve_toeplitz.pyx has not changed
scipy\ndimage\src_cytest.pyx has not changed
scipy\ndimage\src_ni_label.pyx has not changed
scipy\optimize_bglu_dense.pyx has not changed
scipy\optimize_group_columns.pyx has not changed
scipy\linalg_decomp_update.pyx.in has not changed
scipy\optimize_highs\cython\src_highs_constants.pyx has not changed
scipy\optimize_highs\cython\src_highs_wrapper.pyx has not changed
scipy\optimize_lsq\givens_elimination.pyx has not changed
scipy\optimize_trlib_trlib.pyx has not changed
scipy\signal_max_len_seq_inner.pyx has not changed
scipy\signal_peak_finding_utils.pyx has not changed
scipy\signal_sosfilt.pyx has not changed
scipy\signal_spectral.pyx has not changed
scipy\signal_upfirdn_apply.pyx has not changed
scipy\sparse_csparsetools.pyx.in has not changed
scipy\sparse\csgraph_flow.pyx has not changed
scipy\sparse\csgraph_matching.pyx has not changed
scipy\sparse\csgraph_min_spanning_tree.pyx has not changed
scipy\sparse\csgraph_reordering.pyx has not changed
scipy\sparse\csgraph_shortest_path.pyx has not changed
scipy\sparse\csgraph_tools.pyx has not changed
scipy\sparse\csgraph_traversal.pyx has not changed
scipy\spatial\ckdtree.pyx has not changed
scipy\spatial\qhull.pyx has not changed
scipy\spatial_hausdorff.pyx has not changed
scipy\spatial_voronoi.pyx has not changed
scipy\spatial\transform\rotation.pyx has not changed
scipy\special\cython_special.pyx has not changed
scipy\special_comb.pyx has not changed
scipy\special_ellip_harm_2.pyx has not changed
scipy\special_test_round.pyx has not changed
scipy\special_ufuncs.pyx has not changed
scipy\special_ufuncs_cxx.pyx has not changed
scipy\stats\biasedurn.pyx has not changed
scipy\stats_qmc_cy.pyx has not changed
scipy\stats_sobol.pyx has not changed
scipy\stats_stats.pyx has not changed
scipy_lib\messagestream.pyx has not changed
scipy_lib_ccallback_c.pyx has not changed
scipy_lib_test_deprecation_call.pyx has not changed
scipy_lib_test_deprecation_def.pyx has not changed
scipy\io\matlab\mio5_utils.pyx has not changed
scipy\cluster_optimal_leaf_ordering.pyx has not changed
scipy\cluster_vq.pyx has not changed
scipy\fftpack\convolve.pyx has not changed
scipy\interpolate\interpnd.pyx has not changed
scipy\optimize\cython_optimize_zeros.pyx.in has not changed
scipy\interpolate_ppoly.pyx has not changed
Cythonizing sources
lapack_opt_info:
lapack_mkl_info:
No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
customize MSVCCompiler
libraries mkl_rt not found in ['G:/install/envs/pyart_env\Library\lib']
NOT AVAILABLE

openblas_lapack_info:
libraries openblas not found in ['G:/install/envs/pyart_env\Library\lib']
get_default_fcompiler: matching types: '['gnu', 'intelv', 'absoft', 'compaqv', 'intelev', 'gnu95', 'g95', 'intelvem', 'intelem', 'flang']'
customize GnuFCompiler
Could not locate executable g77
Could not locate executable f77
customize IntelVisualFCompiler
Could not locate executable ifort
Could not locate executable ifl
customize AbsoftFCompiler
Could not locate executable f90
customize CompaqVisualFCompiler
Could not locate executable DF
customize IntelItaniumVisualFCompiler
Could not locate executable efl
customize Gnu95FCompiler
Could not locate executable gfortran
Could not locate executable f95
customize G95FCompiler
Could not locate executable g95
customize IntelEM64VisualFCompiler
customize IntelEM64TFCompiler
Could not locate executable efort
Could not locate executable efc
customize PGroupFlangCompiler
Could not locate executable flang
don't know how to compile Fortran code on platform 'nt'
NOT AVAILABLE

openblas_clapack_info:
libraries openblas,lapack not found in ['G:/install/envs/pyart_env\Library\lib']
NOT AVAILABLE

flame_info:
libraries flame not found in ['G:/install/envs/pyart_env\Library\lib']
NOT AVAILABLE

atlas_3_10_threads_info:
Setting PTATLAS=ATLAS
libraries lapack_atlas not found in G:/install/envs/pyart_env\Library\lib
libraries tatlas,tatlas not found in G:/install/envs/pyart_env\Library\lib
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
NOT AVAILABLE

atlas_3_10_info:
libraries lapack_atlas not found in G:/install/envs/pyart_env\Library\lib
libraries satlas,satlas not found in G:/install/envs/pyart_env\Library\lib
<class 'numpy.distutils.system_info.atlas_3_10_info'>
NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
libraries lapack_atlas not found in G:/install/envs/pyart_env\Library\lib
libraries ptf77blas,ptcblas,atlas not found in G:/install/envs/pyart_env\Library\lib
<class 'numpy.distutils.system_info.atlas_threads_info'>
NOT AVAILABLE

atlas_info:
libraries lapack_atlas not found in G:/install/envs/pyart_env\Library\lib
libraries f77blas,cblas,atlas not found in G:/install/envs/pyart_env\Library\lib
<class 'numpy.distutils.system_info.atlas_info'>
NOT AVAILABLE

lapack_info:
libraries lapack,blas not found in ['G:/install/envs/pyart_env\Library\lib']
NOT AVAILABLE

lapack_src_info:
NOT AVAILABLE

NOT AVAILABLE

Running from SciPy source directory.
G:\install\envs\pyart_env\lib\site-packages\numpy\distutils\system_info.py:1849: UserWarning:
Lapack (http://www.netlib.org/lapack/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [lapack]) or by setting
the LAPACK environment variable.
return getattr(self, 'calc_info{}'.format(name))()
G:\install\envs\pyart_env\lib\site-packages\numpy\distutils\system_info.py:1849: UserWarning:
Lapack (http://www.netlib.org/lapack/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [lapack_src]) or by setting
the LAPACK_SRC environment variable.
return getattr(self, 'calc_info{}'.format(name))()
Traceback (most recent call last):
File "setup.py", line 619, in
setup_package()
File "setup.py", line 615, in setup_package
setup(**metadata)
File "G:\install\envs\pyart_env\lib\site-packages\numpy\distutils\core.py", line 135, in setup
config = configuration()
File "setup.py", line 518, in configuration
raise NotFoundError(msg)
numpy.distutils.system_info.NotFoundError: No BLAS/LAPACK libraries found.
To build Scipy from sources, BLAS & LAPACK libraries need to be installed.
See site.cfg.example in the Scipy source directory and
https://docs.scipy.org/doc/scipy/reference/building/index.html for details.

Build failed! (0:00:01.512716 elapsed)

peterbell10

The log tells you what's wrong at the bottom:

numpy.distutils.system_info.NotFoundError: No BLAS/LAPACK libraries found.
To build Scipy from sources, BLAS & LAPACK libraries need to be installed.

Perhaps this windows guide will help with installing libraries:
https://docs.scipy.org/doc/scipy/reference/building/windows.html#build-windows

peterbell10 · 2021-04-19T19:41:25Z

scipy/io/matlab/mio5_utils.pyx

@@ -918,7 +916,7 @@ cdef class VarReader5:
            char *n_ptr = names
            int j, dup_no
        for i in range(n_names):
-            name = asstr(PyBytes_FromString(n_ptr))
+            name = PyBytes_FromString(n_ptr).decode('latin1')


Suggested change

name = PyBytes_FromString(n_ptr).decode('latin1')

name = PyUnicode_FromString(n_ptr)

This change is causing an error 'cpython/PyUnicode_FromString.pxd' not found. I was also working on issue #13880, and I am new to cpython. How can I fix this?

peterbell10 · 2021-04-19T19:42:58Z

scipy/io/matlab/mio5.py

@@ -311,7 +309,7 @@ def get_variables(self, variable_names=None):
        mdict['__globals__'] = []
        while not self.end_of_stream():
            hdr, next_position = self.read_var_header()
-            name = asstr(hdr.name)
+            name = (hdr.name).decode('latin1')


Can you remove the unnecessary brackets here, and everywhere else.

Suggested change

name = (hdr.name).decode('latin1')

name = hdr.name.decode('latin1')

peterbell10 · 2021-04-19T19:53:47Z

scipy/io/mmio.py

+                            stream.write((template % (real(aij),
+                                                             imag(aij).encode('latin1'))


Suggested change

stream.write((template % (real(aij),

imag(aij).encode('latin1'))

data = template % (real(aij), imag(aij))

stream.write(data.encode('latin1'))

peterbell10 · 2021-04-19T19:54:07Z

scipy/io/mmio.py

+                            stream.write((template % (real(aij),
+                                                             imag(aij))).encode('latin1')


Suggested change

stream.write((template % (real(aij),

imag(aij))).encode('latin1')

data = template % (real(aij), imag(aij))

stream.write(data.encode('latin1'))

peterbell10 · 2021-04-19T19:54:53Z

scipy/io/mmio.py

+                    stream.write((("%i %i " % (r, c)) +
+                                         (template % d)).encode('latin1'))


Suggested change

stream.write((("%i %i " % (r, c)) +

(template % d)).encode('latin1'))

stream.write((("%i %i " % (r, c)) +

(template % d)).encode('latin1'))

peterbell10 · 2021-04-19T19:55:07Z

scipy/io/mmio.py

+                    stream.write((("%i %i " % (r, c)) +
+                                         (template % (d.real, d.imag))).encode('latin1'))


Suggested change

stream.write((("%i %i " % (r, c)) +

(template % (d.real, d.imag))).encode('latin1'))

stream.write((("%i %i " % (r, c)) +

(template % (d.real, d.imag))).encode('latin1'))

peterbell10 · 2021-05-16T00:46:59Z

Thanks for your contribution @harishsdev but the numpy.compat uses have now been replaced by gh-14009, so closing this PR.

Remove all usages of numpy.compat ref scipy#13880

4dcb9d5

harishsdev mentioned this pull request Apr 18, 2021

Remove all usages of numpy.compat #13880

Closed

rgommers added the maintenance Items related to regular maintenance tasks label Apr 18, 2021

rgommers requested changes Apr 18, 2021

View reviewed changes

Remove all usages of numpy.compat

2a48cc9

harishsdev requested review from peterbell10 and tylerjereddy as code owners April 19, 2021 19:28

peterbell10 requested changes Apr 19, 2021

View reviewed changes

asstr and asbytes modification

83480c3

peterbell10 closed this May 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove all usages of numpy.compat ref #13880 #13889

Remove all usages of numpy.compat ref #13880 #13889

harishsdev commented Apr 18, 2021 •

edited

rgommers left a comment

rgommers Apr 18, 2021

rgommers Apr 18, 2021

harishsdev commented Apr 19, 2021

peterbell10 left a comment

peterbell10 Apr 19, 2021

NamamiShanker May 1, 2021 •

edited

peterbell10 Apr 19, 2021

peterbell10 Apr 19, 2021

peterbell10 Apr 19, 2021

peterbell10 Apr 19, 2021

peterbell10 Apr 19, 2021

peterbell10 commented May 16, 2021

	name = PyBytes_FromString(n_ptr).decode('latin1')
	name = PyUnicode_FromString(n_ptr)

	name = (hdr.name).decode('latin1')
	name = hdr.name.decode('latin1')

		stream.write((template % (real(aij),
		imag(aij).encode('latin1'))

		stream.write((template % (real(aij),
		imag(aij))).encode('latin1')

		stream.write((("%i %i " % (r, c)) +
		(template % d)).encode('latin1'))

		stream.write((("%i %i " % (r, c)) +
		(template % (d.real, d.imag))).encode('latin1'))

Remove all usages of numpy.compat ref #13880 #13889

Remove all usages of numpy.compat ref #13880 #13889

Conversation

harishsdev commented Apr 18, 2021 • edited

Reference issue

What does this implement/fix?

Additional information

rgommers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harishsdev commented Apr 19, 2021

peterbell10 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NamamiShanker May 1, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peterbell10 commented May 16, 2021

harishsdev commented Apr 18, 2021 •

edited

NamamiShanker May 1, 2021 •

edited