Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SimpleControlSolver.solve() is extremely slow for full gain matrix and large # gait cycles #87

Closed
moorepants opened this issue May 20, 2014 · 8 comments
Closed

SimpleControlSolver.solve() is extremely slow for full gain matrix and large # gait cycles #87

moorepants opened this issue May 20, 2014 · 8 comments
Labels
performance

Comments

@moorepants
Copy link
Member

I'm having trouble figuring out what the slow down is. One bottleneck is surely the parameter covariance computations in dtk.process.least_squares_variance(), but even when that computation is skipped it still seems to be slow.
@moorepants
Copy link
Member Author

Here is an IPython session where I load in the huge dense A and b matrices from file and just try the sparse and non-sparse least squares solvers on them. Neither are super slow (the slow down I'm tracking down is on the order of 30 min to 1 hr for .solve() to run). The sparse finishes in 6 seconds and the dense least squares in 1 minute.

IPython 2.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import numpy as np

In [2]: dat = np.load('data/full-gain-id-lstsq-matrices.npz')

In [3]: A = dat['arr_0']; b = dat['arr_1']

In [4]: A.nbytes
Out[4]: 480729600

In [5]: from scipy import sparse

In [6]: A_csr = sparse.csr_matrix(A)

In [7]: A_csr.data.nbytes + A_csr.indptr.nbytes + A_csr.indices.nbytes
Out[7]: 6163204

In [8]: from scipy.sparse import linalg

In [9]: %time linalg.lsqr(A_csr, b)
CPU times: user 6.28 s, sys: 0 ns, total: 6.28 s
Wall time: 6.29 s
Out[9]: 
(array([   6.96172741,    0.27651683,    7.84628481, ..., -115.1758017 ,
          40.88052914,  -53.92268804]),
 2,
 2449,
 1065.697277746995,
 1065.697277746995,
 4579.6088400047665,
 177037.6789712042,
 0.037634876397199007,
 2865.732011296417,
 array([ 0.,  0.,  0., ...,  0.,  0.,  0.]))

In [10]: %time res_sparse = linalg.lsqr(A_csr, b)
CPU times: user 6.3 s, sys: 16.6 ms, total: 6.32 s
Wall time: 6.33 s

In [11]: from numpy import linalg

In [12]: %time res_dense = linalg.lstsq(A, b)
CPU times: user 1min 16s, sys: 8.69 s, total: 1min 24s
Wall time: 1min 12s

In [13]: res_dense[0]
Out[13]: 
array([   6.96411437,    0.27654199,    7.8445127 , ..., -115.05707248,
         40.90934306,  -53.94957026])

In [14]: res_sparse[0]
Out[14]: 
array([   6.96172741,    0.27651683,    7.84628481, ..., -115.1758017 ,
         40.88052914,  -53.92268804])

In [15]: A.shape
Out[15]: (38520, 1560)

In [16]: b.shape
Out[16]: (38520,)

In [17]: A_csr
Out[17]: 
<38520x1560 sparse matrix of type '<type 'numpy.float64'>'
    with 500760 stored elements in Compressed Sparse Row format>

@moorepants
Copy link
Member Author

%prun A, b = solver.form_a_b() shows that np.linalg.matrix_rank() takes almost a minute to compute for the large dense A matrix.

       485412 function calls in 38.031 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1   36.010   36.010   36.055   36.055 linalg.py:1225(svd)
        1    0.782    0.782   37.770   37.770 controlid.py:319(form_a_b)
        1    0.261    0.261   38.031   38.031 <string>:1(<module>)
    12840    0.105    0.000    0.105    0.000 {numpy.core.multiarray.concatenate}
    13164    0.087    0.000    0.087    0.000 {numpy.core.multiarray.zeros}
        1    0.075    0.075   36.231   36.231 linalg.py:1430(matrix_rank)
     1928    0.062    0.000    0.062    0.000 {method 'reduce' of 'numpy.ufunc' objects}
        1    0.045    0.045    0.045    0.045 {method 'astype' of 'numpy.ndarray' objects}
    32747    0.041    0.000    0.041    0.000 {numpy.core.multiarray.array}
        1    0.040    0.040    0.089    0.089 {method 'max' of 'numpy.ndarray' objects}
    25680    0.038    0.000    0.082    0.000 shape_base.py:8(atleast_1d)
    13482    0.033    0.000    0.096    0.000 frame.py:749(iterrows)
        1    0.032    0.032    0.232    0.232 controlid.py:409(form_control_vectors)
     6420    0.030    0.000    0.039    0.000 twodim_base.py:170(eye)
    27606    0.027    0.000    0.057    0.000 {method 'view' of 'numpy.ndarray' objects}
        1    0.026    0.026    0.196    0.196 controlid.py:429(form_sensor_vectors)
    12840    0.022    0.000    0.209    0.000 shape_base.py:230(hstack)
    28893    0.017    0.000    0.017    0.000 {getattr}
     1926    0.015    0.000    0.029    0.000 common.py:1468(_asarray_tuplesafe)
    25680    0.014    0.000    0.034    0.000 numeric.py:462(asanyarray)
      642    0.013    0.000    0.094    0.000 indexing.py:497(_convert_to_indexer)
     1284    0.013    0.000    0.040    0.000 index.py:86(__new__)
    12840    0.012    0.000    0.028    0.000 series.py:532(__array_finalize__)
      642    0.012    0.000    0.022    0.000 indexing.py:1029(_maybe_convert_indices)
        1    0.011    0.011    0.011    0.011 fromnumeric.py:1621(sum)
    46867    0.011    0.000    0.011    0.000 {isinstance}
      642    0.009    0.000    0.141    0.000 frame.py:2963(take)
    17334    0.008    0.000    0.030    0.000 index.py:2665(_ensure_index)
     1284    0.007    0.000    0.007    0.000 {pandas.lib.infer_dtype}
    12840    0.007    0.000    0.015    0.000 series.py:1297(values)
    66769    0.007    0.000    0.007    0.000 {len}
      642    0.006    0.000    0.006    0.000 {method 'get_indexer' of 'pandas.index.IndexEngine' objects}
      642    0.006    0.000    0.249    0.000 frame.py:2011(_getitem_array)
      642    0.006    0.000    0.042    0.000 index.py:770(get_indexer)
     7066    0.006    0.000    0.027    0.000 numeric.py:392(asarray)
      642    0.005    0.000    0.018    0.000 common.py:462(take_nd)
      642    0.005    0.000    0.010    0.000 index.py:247(is_unique)
    25680    0.005    0.000    0.005    0.000 {method 'append' of 'list' objects}
      642    0.005    0.000    0.036    0.000 internals.py:2219(create_block_manager_from_blocks)
     1284    0.004    0.000    0.016    0.000 frame.py:1722(as_matrix)
     6742    0.004    0.000    0.004    0.000 {range}
      642    0.004    0.000    0.027    0.000 index.py:390(take)
      642    0.004    0.000    0.004    0.000 {pandas.algos.take_2d_axis1_float64_float64}
     1284    0.004    0.000    0.012    0.000 common.py:1232(_is_bool_indexer)
      642    0.004    0.000    0.015    0.000 internals.py:989(__init__)
      642    0.004    0.000    0.048    0.000 frame.py:386(__init__)
      642    0.003    0.000    0.004    0.000 internals.py:1227(shape)
     1284    0.003    0.000    0.003    0.000 frame.py:2090(__setattr__)
      642    0.003    0.000    0.042    0.000 frame.py:530(_init_ndarray)
      642    0.003    0.000    0.003    0.000 index.py:847(_possibly_promote)
     9633    0.003    0.000    0.003    0.000 {issubclass}
      642    0.003    0.000    0.009    0.000 internals.py:1231(_verify_integrity)
      642    0.003    0.000    0.005    0.000 common.py:728(_maybe_promote)
     1926    0.003    0.000    0.008    0.000 generic.py:788(_protect_consolidate)
     1284    0.003    0.000    0.010    0.000 generic.py:750(_consolidate_inplace)
     1926    0.003    0.000    0.016    0.000 _methods.py:31(_any)
      642    0.002    0.000    0.006    0.000 internals.py:932(make_block)
      642    0.002    0.000    0.005    0.000 panel.py:365(__getitem__)
      642    0.002    0.000    0.252    0.000 frame.py:1986(__getitem__)
     1284    0.002    0.000    0.003    0.000 generic.py:78(_get_axis_name)
      642    0.002    0.000    0.002    0.000 internals.py:1358(_consolidate_check)
     1926    0.002    0.000    0.018    0.000 {method 'any' of 'numpy.ndarray' objects}
      642    0.002    0.000    0.002    0.000 {numpy.core.multiarray.empty}
      642    0.002    0.000    0.003    0.000 internals.py:37(__init__)
     2570    0.002    0.000    0.008    0.000 index.py:236(values)
     1284    0.002    0.000    0.002    0.000 internals.py:1505(as_matrix)
      642    0.001    0.000    0.002    0.000 generic.py:66(_get_axis_number)
      642    0.001    0.000    0.001    0.000 generic.py:605(__init__)
      642    0.001    0.000    0.001    0.000 {method 'take' of 'numpy.ndarray' objects}
      642    0.001    0.000    0.001    0.000 {method 'get_loc' of 'pandas.index.IndexEngine' objects}
      644    0.001    0.000    0.006    0.000 panel.py:495(iteritems)
     1284    0.001    0.000    0.005    0.000 generic.py:90(_get_axis)
      642    0.001    0.000    0.002    0.000 common.py:431(_get_take_nd_function)
      642    0.001    0.000    0.001    0.000 index.py:269(_engine)
      642    0.001    0.000    0.003    0.000 index.py:189(dtype)
      642    0.001    0.000    0.002    0.000 indexing.py:1077(_is_list_like)
     3213    0.001    0.000    0.001    0.000 {method 'get' of 'dict' objects}
     1284    0.001    0.000    0.002    0.000 common.py:1521(is_integer)
      642    0.001    0.000    0.002    0.000 frame.py:5734(_prep_ndarray)
     1284    0.001    0.000    0.002    0.000 index.py:135(__array_finalize__)
      642    0.001    0.000    0.001    0.000 indexing.py:910(_convert_to_index_sliceable)
     1284    0.001    0.000    0.001    0.000 internals.py:1640(_consolidate_inplace)
     1284    0.001    0.000    0.002    0.000 generic.py:751(<lambda>)
      642    0.001    0.000    0.001    0.000 generic.py:662(_get_item_cache)
      642    0.001    0.000    0.002    0.000 index.py:708(get_loc)
      642    0.001    0.000    0.004    0.000 generic.py:778(_is_mixed_type)
      644    0.001    0.000    0.004    0.000 index.py:293(__iter__)
     2568    0.001    0.000    0.001    0.000 internals.py:1350(is_consolidated)
      642    0.001    0.000    0.002    0.000 internals.py:1363(is_mixed_type)
     1926    0.001    0.000    0.001    0.000 internals.py:1229(<genexpr>)
     1284    0.001    0.000    0.001    0.000 internals.py:1626(consolidate)
      642    0.001    0.000    0.002    0.000 generic.py:780(<lambda>)
      642    0.001    0.000    0.001    0.000 {hasattr}
      642    0.001    0.000    0.001    0.000 generic.py:102(_indexer)
        1    0.001    0.001    0.001    0.001 {max}
      642    0.001    0.000    0.003    0.000 index.py:272(<lambda>)
      642    0.001    0.000    0.001    0.000 indexing.py:1063(_is_integer_dtype)
      642    0.001    0.000    0.001    0.000 index.py:879(_get_method)
      642    0.001    0.000    0.001    0.000 indexing.py:1068(_is_integer_index)
      642    0.001    0.000    0.001    0.000 function_base.py:40(iterable)
      642    0.001    0.000    0.001    0.000 generic.py:659(__getitem__)
      642    0.001    0.000    0.001    0.000 {sum}
     1286    0.000    0.000    0.000    0.000 {iter}
      642    0.000    0.000    0.001    0.000 common.py:1525(is_float)
     1926    0.000    0.000    0.000    0.000 internals.py:1177(_get_items)
     1284    0.000    0.000    0.001    0.000 internals.py:1233(<genexpr>)
      642    0.000    0.000    0.000    0.000 internals.py:76(set_ref_locs)
      642    0.000    0.000    0.000    0.000 base.py:45(_constructor)
     1284    0.000    0.000    0.000    0.000 {pandas.algos.ensure_platform_int}
     1284    0.000    0.000    0.000    0.000 {pandas.algos.ensure_int64}
      642    0.000    0.000    0.000    0.000 internals.py:128(dtype)
      321    0.000    0.000    0.000    0.000 controlid.py:120(sensors)
      321    0.000    0.000    0.000    0.000 controlid.py:89(controls)
        1    0.000    0.000    0.000    0.000 getlimits.py:93(__new__)
        1    0.000    0.000    0.048    0.048 _methods.py:15(_amax)
        2    0.000    0.000    0.000    0.000 linalg.py:124(_realType)
        1    0.000    0.000    0.000    0.000 _methods.py:23(_sum)
        1    0.000    0.000    0.000    0.000 linalg.py:139(_commonType)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        2    0.000    0.000    0.000    0.000 controlid.py:98(identification_data)
        1    0.000    0.000    0.000    0.000 linalg.py:106(_makearray)
        1    0.000    0.000    0.000    0.000 linalg.py:101(get_linalg_error_extobj)
        2    0.000    0.000    0.000    0.000 linalg.py:111(isComplexType)
        1    0.000    0.000    0.000    0.000 linalg.py:198(_assertRankAtLeast2)
        1    0.000    0.000    0.000    0.000 linalg.py:219(_assertNoEmpty2d)
        1    0.000    0.000    0.000    0.000 controlid.py:108(gain_omission_matrix)

@moorepants
Copy link
Member Author

This shows the profiling for the %prun x, variance, covariance = solver.least_squares(A, b, ignore_cov=True) command. It only took 32 seconds! So this is not the issue either.

         82 function calls in 32.407 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        2   24.514   12.257   24.514   12.257 {numpy.linalg.lapack_lite.dgelsd}
        1    3.269    3.269   30.887   30.887 linalg.py:1733(lstsq)
        6    1.895    0.316    1.895    0.316 {numpy.core.multiarray.zeros}
        2    1.292    0.646    1.292    0.646 {numpy.core.multiarray._fastCopyAndTranspose}
        1    0.781    0.781    0.781    0.781 {math.log}
        2    0.287    0.143    0.287    0.143 {max}
        2    0.102    0.051    0.102    0.051 {method 'copy' of 'numpy.ndarray' objects}
        2    0.099    0.049    0.102    0.051 fromnumeric.py:1281(ravel)
        7    0.095    0.014    0.095    0.014 {numpy.core.multiarray.array}
        1    0.071    0.071   32.407   32.407 controlid.py:448(least_squares)
        2    0.001    0.001    0.001    0.001 {method 'ravel' of 'numpy.ndarray' objects}
        1    0.001    0.001    0.001    0.001 {method 'reduce' of 'numpy.ufunc' objects}
        1    0.001    0.001    0.002    0.002 fromnumeric.py:1621(sum)
        6    0.001    0.000    0.001    0.000 {issubclass}
        1    0.000    0.000   32.407   32.407 <string>:1(<module>)
        1    0.000    0.000    1.292    1.292 linalg.py:180(_fastCopyAndTranspose)
        4    0.000    0.000    0.003    0.001 numeric.py:392(asarray)
        1    0.000    0.000    0.000    0.000 linalg.py:139(_commonType)
        4    0.000    0.000    0.001    0.000 linalg.py:111(isComplexType)
        1    0.000    0.000    0.000    0.000 linalg.py:168(_to_native_byte_order)
        1    0.000    0.000    0.001    0.001 _methods.py:23(_sum)
        5    0.000    0.000    0.000    0.000 {min}
        2    0.000    0.000    0.000    0.000 linalg.py:106(_makearray)
        2    0.000    0.000    0.000    0.000 {isinstance}
        3    0.000    0.000    0.000    0.000 linalg.py:124(_realType)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 base.py:702(isspmatrix)
        7    0.000    0.000    0.000    0.000 {len}
        1    0.000    0.000    0.000    0.000 {method 'astype' of 'numpy.ndarray' objects}
        1    0.000    0.000    0.000    0.000 linalg.py:192(_assertRank2)
        2    0.000    0.000    0.000    0.000 {getattr}
        3    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
        2    0.000    0.000    0.000    0.000 {method '__array_prepare__' of 'numpy.ndarray' objects}
        2    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 linalg.py:130(_linalgRealType)

@moorepants
Copy link
Member Author

%prun deconstructed_solution = solver.deconstruct_solution(x, covariance)

         55 function calls in 0.217 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.125    0.125    0.125    0.125 {method 'diagonal' of 'numpy.ndarray' objects}
        1    0.058    0.058    0.183    0.183 controlid.py:231(deconstruct_solution)
        1    0.034    0.034    0.217    0.217 <string>:1(<module>)
        4    0.000    0.000    0.000    0.000 {numpy.core.multiarray.zeros}
        1    0.000    0.000    0.125    0.125 twodim_base.py:221(diag)
       40    0.000    0.000    0.000    0.000 {method 'reshape' of 'numpy.ndarray' objects}
        1    0.000    0.000    0.000    0.000 numeric.py:392(asarray)
        1    0.000    0.000    0.000    0.000 {numpy.core.multiarray.array}
        1    0.000    0.000    0.000    0.000 {range}
        1    0.000    0.000    0.000    0.000 controlid.py:108(gain_omission_matrix)
        2    0.000    0.000    0.000    0.000 {len}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

@moorepants
Copy link
Member Author

%prun estimated_controls = solver.compute_estimated_controls(gain_matrices, nominal_controls)

         8825091 function calls (8721303 primitive calls) in 15.053 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    78628    0.894    0.000    0.894    0.000 {method 'reduce' of 'numpy.ufunc' objects}
   246038    0.761    0.000    0.761    0.000 {numpy.core.multiarray.array}
    69229    0.709    0.000    5.358    0.000 indexing.py:497(_convert_to_indexer)
    32400    0.665    0.000    2.016    0.000 indexing.py:100(_setitem_with_indexer)
        1    0.611    0.611   15.053   15.053 controlid.py:142(compute_estimated_controls)
  2371345    0.587    0.000    0.587    0.000 {isinstance}
      108    0.584    0.005    0.584    0.005 {numpy.core.multiarray.zeros}
    78520    0.580    0.000    1.157    0.000 common.py:1468(_asarray_tuplesafe)
      325    0.475    0.001    0.475    0.001 {method 'copy' of 'numpy.ndarray' objects}
    41692    0.373    0.000    1.229    0.000 index.py:86(__new__)
    25920    0.352    0.000    2.630    0.000 indexing.py:332(_getitem_lowerdim)
    36829    0.349    0.000    0.349    0.000 {method 'get_indexer' of 'pandas.index.IndexEngine' objects}
    36829    0.319    0.000    2.384    0.000 index.py:770(get_indexer)
        1    0.299    0.299    0.319    0.319 common.py:134(_isnull_ndarraylike)
    46012    0.278    0.000    0.302    0.000 {pandas.lib.infer_dtype}
    32400    0.262    0.000    4.999    0.000 indexing.py:93(_convert_tuple)
   165138    0.261    0.000    0.404    0.000 {method 'view' of 'numpy.ndarray' objects}
    30240    0.259    0.000    1.450    0.000 frame.py:1928(_ixs)
    30240    0.250    0.000    0.815    0.000 series.py:430(__new__)
   245712    0.240    0.000    1.000    0.000 numeric.py:392(asarray)
   149259    0.229    0.000    0.562    0.000 common.py:1232(_is_bool_indexer)
409650/357810    0.193    0.000    0.287    0.000 {getattr}
   132196    0.175    0.000    0.241    0.000 generic.py:78(_get_axis_name)
    36829    0.162    0.000    0.162    0.000 index.py:847(_possibly_promote)
    32400    0.160    0.000    7.313    0.000 indexing.py:69(__setitem__)
    32508    0.139    0.000    0.659    0.000 fromnumeric.py:2249(prod)
    92989    0.137    0.000    0.290    0.000 generic.py:102(_indexer)
    88669    0.137    0.000    0.563    0.000 indexing.py:1077(_is_list_like)
    30240    0.129    0.000    0.275    0.000 series.py:3299(_sanitize_array)
   126057    0.128    0.000    1.322    0.000 index.py:2665(_ensure_index)
    69337    0.128    0.000    0.128    0.000 {method 'get_loc' of 'pandas.index.IndexEngine' objects}
    56160    0.122    0.000    1.822    0.000 indexing.py:835(_getitem_axis)
   181659    0.117    0.000    0.170    0.000 common.py:1521(is_integer)
    30240    0.112    0.000    0.116    0.000 internals.py:1600(fast_2d_xs)
    37153    0.112    0.000    0.431    0.000 frame.py:1722(as_matrix)
   345087    0.111    0.000    0.111    0.000 {issubclass}
   132194    0.109    0.000    0.408    0.000 generic.py:90(_get_axis)
    74415    0.108    0.000    0.802    0.000 generic.py:788(_protect_consolidate)
    62641    0.108    0.000    0.250    0.000 index.py:332(__getitem__)
    41906    0.106    0.000    0.131    0.000 frame.py:2090(__setattr__)
    88669    0.106    0.000    0.391    0.000 function_base.py:40(iterable)
114808/88780    0.101    0.000    0.286    0.000 {iter}
    25920    0.097    0.000    0.390    0.000 indexing.py:687(_has_valid_tuple)
     4646    0.093    0.000    0.123    0.000 index.py:247(is_unique)
    30240    0.084    0.000    0.084    0.000 common.py:1135(_possibly_castable)
     4429    0.080    0.000    0.152    0.000 indexing.py:1029(_maybe_convert_indices)
    51840    0.078    0.000    0.218    0.000 indexing.py:794(_has_valid_type)
    37262    0.071    0.000    0.764    0.000 generic.py:750(_consolidate_inplace)
   581220    0.070    0.000    0.070    0.000 {len}
    41258    0.068    0.000    0.225    0.000 index.py:189(dtype)
    26028    0.066    0.000    0.194    0.000 series.py:1210(__iter__)
    69337    0.066    0.000    0.193    0.000 index.py:708(get_loc)
    82405    0.064    0.000    0.114    0.000 generic.py:651(ndim)
56160/30240    0.062    0.000    3.400    0.000 indexing.py:695(__getitem__)
    45688    0.062    0.000    0.458    0.000 _methods.py:31(_any)
     4429    0.061    0.000    1.019    0.000 frame.py:2963(take)
    56160    0.060    0.000    1.558    0.000 indexing.py:63(_get_loc)
    62751    0.057    0.000    0.075    0.000 numeric.py:1810(isscalar)
    32400    0.055    0.000    0.083    0.000 indexing.py:1043(_maybe_convert_ix)
    82952    0.052    0.000    0.258    0.000 index.py:236(values)
    45688    0.050    0.000    0.509    0.000 {method 'any' of 'numpy.ndarray' objects}
    32508    0.049    0.000    0.520    0.000 _methods.py:27(_prod)
      108    0.049    0.000    0.049    0.000 {pandas.lib.clean_index_list}
    36940    0.048    0.000    0.048    0.000 {hasattr}
    37153    0.046    0.000    0.244    0.000 generic.py:778(_is_mixed_type)
    37154    0.044    0.000    0.050    0.000 internals.py:1505(as_matrix)
    30240    0.043    0.000    0.092    0.000 series.py:532(__array_finalize__)
    26028    0.043    0.000    0.074    0.000 numerictypes.py:735(issubdtype)
    69229    0.043    0.000    0.044    0.000 indexing.py:1068(_is_integer_index)
    25920    0.041    0.000    3.062    0.000 indexing.py:804(_getitem_tuple)
   177995    0.041    0.000    0.041    0.000 {method 'get' of 'dict' objects}
    37153    0.040    0.000    0.087    0.000 internals.py:1363(is_mixed_type)
    82513    0.040    0.000    0.050    0.000 internals.py:1016(ndim)
     4429    0.039    0.000    0.039    0.000 {pandas.algos.take_2d_axis1_float64_float64}
     4429    0.038    0.000    0.155    0.000 common.py:462(take_nd)
    37153    0.038    0.000    0.125    0.000 generic.py:780(<lambda>)
      216    0.037    0.000    0.037    0.000 generic.py:662(_get_item_cache)
     4429    0.037    0.000    1.760    0.000 frame.py:2011(_getitem_array)
     4755    0.035    0.000    0.128    0.000 internals.py:989(__init__)
     4646    0.034    0.000    0.284    0.000 internals.py:2219(create_block_manager_from_blocks)
    41908    0.034    0.000    0.051    0.000 index.py:135(__array_finalize__)
    36829    0.033    0.000    0.048    0.000 indexing.py:1063(_is_integer_dtype)
     4430    0.032    0.000    0.032    0.000 {numpy.core.multiarray.empty}
    36829    0.032    0.000    0.041    0.000 index.py:879(_get_method)
    30240    0.031    0.000    0.115    0.000 series.py:3310(_try_cast)
    41800    0.031    0.000    0.046    0.000 internals.py:1640(_consolidate_inplace)
     4320    0.030    0.000    0.030    0.000 {numpy.core.multiarray.dot}
     4862    0.028    0.000    0.456    0.000 frame.py:386(__init__)
    30565    0.028    0.000    0.035    0.000 indexing.py:32(__init__)
    25920    0.028    0.000    0.028    0.000 {pandas.index.get_value_at}
     4429    0.027    0.000    0.184    0.000 index.py:390(take)
     4863    0.027    0.000    0.034    0.000 internals.py:1227(shape)
    37262    0.027    0.000    0.552    0.000 generic.py:751(<lambda>)
    25920    0.027    0.000    0.143    0.000 indexing.py:1072(_is_label_like)
    79062    0.026    0.000    0.026    0.000 internals.py:1350(is_consolidated)
     4646    0.025    0.000    0.412    0.000 frame.py:530(_init_ndarray)
     4755    0.024    0.000    0.069    0.000 internals.py:1231(_verify_integrity)
    26136    0.023    0.000    0.046    0.000 series.py:1297(values)
    37262    0.022    0.000    0.525    0.000 internals.py:1626(consolidate)
      324    0.022    0.000    0.022    0.000 {numpy.core.multiarray.concatenate}
     4971    0.021    0.000    0.048    0.000 internals.py:932(make_block)
    25920    0.021    0.000    0.048    0.000 series.py:578(_ixs)
     5077    0.020    0.000    0.021    0.000 index.py:318(__contains__)
    65845    0.020    0.000    0.020    0.000 {method 'append' of 'list' objects}
     4429    0.019    0.000    0.032    0.000 common.py:728(_maybe_promote)
     4537    0.016    0.000    1.790    0.000 frame.py:1986(__getitem__)
     4863    0.016    0.000    0.019    0.000 internals.py:1358(_consolidate_check)
     4971    0.015    0.000    0.025    0.000 internals.py:37(__init__)
    30565    0.015    0.000    0.057    0.000 {setattr}
    26028    0.015    0.000    0.024    0.000 numerictypes.py:667(issubclass_)
    28084    0.015    0.000    0.015    0.000 controlid.py:89(controls)
        1    0.012    0.012    0.331    0.331 common.py:64(_isnull_new)
     4863    0.012    0.000    0.012    0.000 generic.py:605(__init__)
    41258    0.011    0.000    0.011    0.000 {pandas.algos.ensure_platform_int}
     4429    0.011    0.000    0.011    0.000 {method 'take' of 'numpy.ndarray' objects}
     4430    0.010    0.000    0.016    0.000 generic.py:66(_get_axis_number)
     4646    0.009    0.000    0.010    0.000 index.py:269(_engine)
     4429    0.009    0.000    0.013    0.000 common.py:431(_get_take_nd_function)
     4645    0.009    0.000    0.012    0.000 indexing.py:910(_convert_to_index_sliceable)
     4646    0.007    0.000    0.041    0.000 frame.py:5734(_prep_ndarray)
    30240    0.006    0.000    0.006    0.000 series.py:521(__init__)
    14590    0.006    0.000    0.007    0.000 internals.py:1229(<genexpr>)
     4321    0.005    0.000    0.060    0.000 index.py:289(is_all_dates)
     4646    0.005    0.000    0.021    0.000 index.py:272(<lambda>)
     4755    0.004    0.000    0.008    0.000 {sum}
    15669    0.004    0.000    0.004    0.000 internals.py:1177(_get_items)
     4429    0.004    0.000    0.005    0.000 common.py:1525(is_float)
      430    0.004    0.000    0.011    0.000 numeric.py:2222(array_equal)
     6595    0.004    0.000    0.004    0.000 controlid.py:120(sensors)
     4429    0.003    0.000    0.056    0.000 index.py:282(inferred_type)
     9510    0.003    0.000    0.004    0.000 internals.py:1233(<genexpr>)
     4971    0.003    0.000    0.003    0.000 internals.py:76(set_ref_locs)
     4537    0.002    0.000    0.002    0.000 base.py:45(_constructor)
     5403    0.002    0.000    0.002    0.000 internals.py:128(dtype)
     8858    0.002    0.000    0.002    0.000 {pandas.algos.ensure_int64}
      108    0.002    0.000    0.025    0.000 frame.py:3818(_combine_frame)
      324    0.001    0.000    0.002    0.000 shape_base.py:60(atleast_2d)
      108    0.001    0.000    0.490    0.005 internals.py:2471(_consolidate)
      108    0.001    0.000    0.087    0.001 internals.py:1722(set)
      108    0.001    0.000    0.005    0.000 frame.py:2183(_sanitize_column)
      108    0.001    0.000    0.084    0.001 internals.py:1785(insert)
      108    0.001    0.000    0.007    0.000 index.py:347(append)
      755    0.001    0.000    0.012    0.000 index.py:450(equals)
     5077    0.001    0.000    0.001    0.000 {hash}
      108    0.001    0.000    0.003    0.000 internals.py:1904(_add_new_block)
      109    0.001    0.000    0.001    0.000 {pandas.lib.list_to_object_array}
      108    0.001    0.000    0.028    0.000 index.py:1272(insert)
      108    0.001    0.000    0.001    0.000 {operator.sub}
      108    0.001    0.000    0.001    0.000 index.py:378(_ensure_compat_concat)
        1    0.001    0.001    0.507    0.507 panel.py:273(_init_dict)
      108    0.001    0.000    0.488    0.005 internals.py:2489(_merge_blocks)
      108    0.001    0.000    0.006    0.000 frame.py:2499(_align_frame)
      108    0.001    0.000    0.052    0.000 internals.py:1020(set_axis)
      108    0.001    0.000    0.003    0.000 expressions.py:58(_can_use_numexpr)
      216    0.001    0.000    0.051    0.000 internals.py:1072(_set_ref_locs)
      108    0.001    0.000    0.006    0.000 frame.py:194(na_op)
      108    0.001    0.000    0.088    0.001 generic.py:678(_set_item)
        1    0.001    0.001    0.001    0.001 internals.py:2368(_stack_arrays)
      430    0.001    0.000    0.004    0.000 _methods.py:35(_all)
      108    0.001    0.000    0.038    0.000 panel.py:365(__getitem__)
      108    0.001    0.000    0.476    0.004 internals.py:158(reindex_items_from)
      108    0.001    0.000    0.025    0.000 frame.py:215(f)
      109    0.000    0.000    0.001    0.000 {sorted}
      108    0.000    0.000    0.093    0.001 frame.py:2155(_set_item)
      430    0.000    0.000    0.004    0.000 {method 'all' of 'numpy.ndarray' objects}
      108    0.000    0.000    0.500    0.005 frame.py:2573(reindex)
      324    0.000    0.000    0.007    0.000 index.py:889(reindex)
      109    0.000    0.000    0.039    0.000 panel.py:495(iteritems)
      108    0.000    0.000    0.094    0.001 frame.py:2107(__setitem__)
        1    0.000    0.000    0.000    0.000 {numpy.core.multiarray.putmask}
      108    0.000    0.000    0.005    0.000 expressions.py:82(_evaluate_numexpr)
      216    0.000    0.000    0.002    0.000 frame.py:2717(_reindex_with_indexers)
      108    0.000    0.000    0.001    0.000 common.py:1614(_is_sequence)
      108    0.000    0.000    0.005    0.000 expressions.py:145(evaluate)
      108    0.000    0.000    0.000    0.000 internals.py:83(set_ref_items)
      324    0.000    0.000    0.001    0.000 numeric.py:462(asanyarray)
      108    0.000    0.000    0.002    0.000 shape_base.py:179(vstack)
      108    0.000    0.000    0.006    0.000 frame.py:2453(align)
      108    0.000    0.000    0.004    0.000 frame.py:2681(_reindex_multi)
      108    0.000    0.000    0.052    0.000 internals.py:1816(set_items_norename)
        2    0.000    0.000    0.003    0.001 panel.py:1539(_extract_axis)
      108    0.000    0.000    0.000    0.000 {range}
      108    0.000    0.000    0.006    0.000 frame.py:3822(_arith_op)
      216    0.000    0.000    0.000    0.000 internals.py:2475(<lambda>)
      108    0.000    0.000    0.003    0.000 internals.py:2521(_vstack)
      108    0.000    0.000    0.000    0.000 generic.py:675(_clear_item_cache)
      108    0.000    0.000    0.037    0.000 generic.py:659(__getitem__)
      108    0.000    0.000    0.001    0.000 expressions.py:54(_evaluate_standard)
        1    0.000    0.000    0.001    0.001 internals.py:2254(form_blocks)
      216    0.000    0.000    0.000    0.000 internals.py:2479(<lambda>)
      216    0.000    0.000    0.000    0.000 frame.py:466(_init_mgr)
      108    0.000    0.000    0.000    0.000 internals.py:2372(_asarray_compat)
        1    0.000    0.000    0.358    0.358 nanops.py:207(nanmean)
        1    0.000    0.000    0.000    0.000 {pandas.lib.isscalar}
      108    0.000    0.000    0.000    0.000 internals.py:2513(_block_shape)
        1    0.000    0.000    0.002    0.002 index.py:2708(_union_indexes)
      108    0.000    0.000    0.000    0.000 common.py:966(_fill_zeros)
        2    0.000    0.000    0.000    0.000 index.py:2704(_get_distinct_indexes)
        1    0.000    0.000    0.507    0.507 panel.py:236(_init_data)
        1    0.000    0.000    0.010    0.010 nanops.py:447(_get_counts)
        1    0.000    0.000    0.000    0.000 {pandas.lib.ismember}
      108    0.000    0.000    0.000    0.000 {method 'copy' of 'dict' objects}
        4    0.000    0.000    0.000    0.000 {zip}
        1    0.000    0.000    0.335    0.335 nanops.py:125(_get_values)
        1    0.000    0.000    0.001    0.001 internals.py:2351(_multi_blockify)
        1    0.000    0.000    0.000    0.000 index.py:860(isin)
      109    0.000    0.000    0.012    0.000 {any}
      218    0.000    0.000    0.000    0.000 index.py:2705(<genexpr>)
        1    0.000    0.000    0.000    0.000 index.py:2749(_sanitize_and_check)
      216    0.000    0.000    0.000    0.000 {id}
        2    0.000    0.000    0.000    0.000 {method 'astype' of 'numpy.ndarray' objects}
      108    0.000    0.000    0.000    0.000 {method 'clear' of 'dict' objects}
        1    0.000    0.000   15.053   15.053 <string>:1(<module>)
        3    0.000    0.000    0.012    0.004 nanops.py:26(check)
        2    0.000    0.000    0.000    0.000 panel.py:1475(_extract_axes_for_slice)
      108    0.000    0.000    0.000    0.000 internals.py:2355(<lambda>)
        1    0.000    0.000    0.370    0.370 panel.py:1259(_reduce)
        1    0.000    0.000    0.001    0.001 internals.py:2236(create_block_manager_from_arrays)
        1    0.000    0.000    0.370    0.370 nanops.py:31(_f)
        2    0.000    0.000    0.023    0.012 _methods.py:23(_sum)
        1    0.000    0.000    0.000    0.000 panel.py:503(_get_plane_axes)
        2    0.000    0.000    0.023    0.012 {method 'sum' of 'numpy.ndarray' objects}
        1    0.000    0.000    0.000    0.000 nanops.py:543(_ensure_numeric)
        1    0.000    0.000    0.358    0.358 nanops.py:55(f)
        1    0.000    0.000    0.000    0.000 nanops.py:161(_na_ok_dtype)
        4    0.000    0.000    0.012    0.003 nanops.py:34(<genexpr>)
        3    0.000    0.000    0.000    0.000 index.py:293(__iter__)
        2    0.000    0.000    0.000    0.000 index.py:1341(__new__)
        2    0.000    0.000    0.002    0.001 index.py:2689(_get_combined_index)
        1    0.000    0.000    0.000    0.000 panel.py:572(_get_values)
        1    0.000    0.000    0.370    0.370 panel.py:1642(mean)
        1    0.000    0.000    0.000    0.000 nanops.py:164(_view_if_needed)
        1    0.000    0.000    0.507    0.507 panel.py:230(__init__)
        1    0.000    0.000    0.000    0.000 {method 'keys' of 'dict' objects}
        4    0.000    0.000    0.000    0.000 {method 'values' of 'dict' objects}
        1    0.000    0.000    0.001    0.001 panel.py:316(_init_arrays)
        1    0.000    0.000    0.000    0.000 common.py:1292(_try_sort)
        1    0.000    0.000    0.370    0.370 panel.py:1262(<lambda>)
        1    0.000    0.000    0.331    0.331 common.py:47(isnull)
        2    0.000    0.000    0.000    0.000 controlid.py:129(validation_data)
        1    0.000    0.000    0.000    0.000 internals.py:2379(_shape_compat)
        1    0.000    0.000    0.000    0.000 {method 'pop' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'itervalues' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'iteritems' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 nanops.py:103(_get_fill_value)
        1    0.000    0.000    0.000    0.000 {method 'extend' of 'list' objects}
        1    0.000    0.000    0.000    0.000 index.py:876(_array_values)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

@moorepants
Copy link
Member Author

The above shows the profile for each function call inside .solve() and the total should complete in under a couple of minutes. But it wasn't doing that before...

@moorepants
Copy link
Member Author

Ok I reran %prun result = solver.solve(ignore_cov=True) and it seems as if everything is computing in a reasonable time. The major slow down is the parameter covariance computations for a large A matrix. Now below shows that the matrix_rank checks are the main unnecessary speed bottlenecks.

         9310636 function calls (9206848 primitive calls) in 115.147 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1   62.446   62.446   64.255   64.255 linalg.py:1225(svd)
        2   22.596   11.298   22.596   11.298 {numpy.linalg.lapack_lite.dgelsd}
        1    2.759    2.759   28.948   28.948 linalg.py:1733(lstsq)
        4    1.814    0.454    1.814    0.454 {method 'astype' of 'numpy.ndarray' objects}
        2    1.247    0.624    1.247    0.624 {numpy.core.multiarray._fastCopyAndTranspose}
    80557    1.101    0.000    1.101    0.000 {method 'reduce' of 'numpy.ufunc' objects}
   278793    1.053    0.000    1.053    0.000 {numpy.core.multiarray.array}
        1    1.037    1.037    1.037    1.037 {math.log}
        1    1.029    1.029    2.276    2.276 linalg.py:180(_fastCopyAndTranspose)
    80446    0.762    0.000    1.497    0.000 common.py:1468(_asarray_tuplesafe)
   192744    0.752    0.000    0.915    0.000 {method 'view' of 'numpy.ndarray' objects}
        1    0.735    0.735   65.938   65.938 linalg.py:1430(matrix_rank)
    13164    0.715    0.000    0.715    0.000 {numpy.core.multiarray.concatenate}
        1    0.714    0.714   17.476   17.476 controlid.py:142(compute_estimated_controls)
    69871    0.703    0.000    6.998    0.000 indexing.py:497(_convert_to_indexer)
    32400    0.635    0.000    1.869    0.000 indexing.py:100(_setitem_with_indexer)
   252783    0.610    0.000    1.643    0.000 numeric.py:392(asarray)
  2418214    0.601    0.000    0.601    0.000 {isinstance}
        1    0.570    0.570   67.831   67.831 controlid.py:319(form_a_b)
        2    0.550    0.275    0.825    0.412 fromnumeric.py:1621(sum)
     5613    0.513    0.000    0.541    0.000 internals.py:932(make_block)
    37471    0.503    0.000    0.503    0.000 {method 'get_indexer' of 'pandas.index.IndexEngine' objects}
    42976    0.361    0.000    1.766    0.000 index.py:86(__new__)
    13282    0.335    0.000    0.335    0.000 {numpy.core.multiarray.zeros}
    25920    0.331    0.000    2.464    0.000 indexing.py:332(_getitem_lowerdim)
    37471    0.306    0.000    3.559    0.000 index.py:770(get_indexer)
    32400    0.289    0.000    5.976    0.000 indexing.py:93(_convert_tuple)
        1    0.275    0.275  115.030  115.030 controlid.py:719(solve)
    30240    0.272    0.000    0.795    0.000 series.py:430(__new__)
    47296    0.267    0.000    0.290    0.000 {pandas.lib.infer_dtype}
    38976    0.254    0.000    0.282    0.000 series.py:1297(values)
    42542    0.245    0.000    0.245    0.000 {pandas.algos.ensure_platform_int}
    30240    0.236    0.000    1.375    0.000 frame.py:1928(_ixs)
438545/386705    0.229    0.000    0.317    0.000 {getattr}
     5071    0.227    0.000    0.227    0.000 {pandas.algos.take_2d_axis1_float64_float64}
   150543    0.219    0.000    0.750    0.000 common.py:1232(_is_bool_indexer)
    69979    0.210    0.000    0.210    0.000 {method 'get_loc' of 'pandas.index.IndexEngine' objects}
      108    0.200    0.002    0.200    0.002 {pandas.lib.clean_index_list}
   133480    0.166    0.000    0.229    0.000 generic.py:78(_get_axis_name)
    37471    0.155    0.000    0.155    0.000 index.py:847(_possibly_promote)
        1    0.152    0.152    0.152    0.152 {method 'itervalues' of 'dict' objects}
        1    0.150    0.150    0.150    0.150 linalg.py:168(_to_native_byte_order)
    32400    0.149    0.000    8.124    0.000 indexing.py:69(__setitem__)
        1    0.149    0.149   29.162   29.162 controlid.py:448(least_squares)
     5288    0.148    0.000    0.181    0.000 index.py:247(is_unique)
    32508    0.144    0.000    0.601    0.000 fromnumeric.py:2249(prod)
        1    0.135    0.135    0.146    0.146 {pandas.lib.isscalar}
    89311    0.129    0.000    0.536    0.000 indexing.py:1077(_is_list_like)
    93631    0.129    0.000    0.378    0.000 generic.py:102(_indexer)
   143391    0.129    0.000    1.947    0.000 index.py:2665(_ensure_index)
        2    0.122    0.061    0.122    0.061 {method 'ravel' of 'numpy.ndarray' objects}
    38437    0.120    0.000    0.471    0.000 frame.py:1722(as_matrix)
    76341    0.117    0.000    0.404    0.000 generic.py:788(_protect_consolidate)
    30240    0.117    0.000    0.252    0.000 series.py:3299(_sanitize_array)
    56160    0.114    0.000    1.742    0.000 indexing.py:835(_getitem_axis)
   182943    0.110    0.000    0.161    0.000 common.py:1521(is_integer)
   354726    0.107    0.000    0.107    0.000 {issubclass}
   133478    0.106    0.000    0.390    0.000 generic.py:90(_get_axis)
    30240    0.104    0.000    0.107    0.000 internals.py:1600(fast_2d_xs)
116094/90066    0.102    0.000    0.500    0.000 {iter}
    89311    0.101    0.000    0.374    0.000 function_base.py:40(iterable)
     5071    0.101    0.000    0.400    0.000 common.py:462(take_nd)
    43190    0.100    0.000    0.156    0.000 frame.py:2090(__setattr__)
      858    0.099    0.000    0.099    0.000 generic.py:662(_get_item_cache)
    62641    0.099    0.000    0.228    0.000 index.py:332(__getitem__)
    38438    0.094    0.000    0.100    0.000 internals.py:1505(as_matrix)
     5071    0.090    0.000    2.998    0.001 frame.py:2011(_getitem_array)
    25920    0.089    0.000    0.364    0.000 indexing.py:687(_has_valid_tuple)
        1    0.088    0.088    0.088    0.088 common.py:134(_isnull_ndarraylike)
     5505    0.087    0.000    0.087    0.000 generic.py:605(__init__)
     5071    0.084    0.000    0.169    0.000 indexing.py:1029(_maybe_convert_indices)
        1    0.084    0.084    0.285    0.285 controlid.py:231(deconstruct_solution)
    30240    0.078    0.000    0.078    0.000 common.py:1135(_possibly_castable)
      108    0.077    0.001    0.092    0.001 internals.py:2471(_consolidate)
    51840    0.074    0.000    0.205    0.000 indexing.py:794(_has_valid_type)
   181211    0.074    0.000    0.074    0.000 {method 'get' of 'dict' objects}
   647998    0.071    0.000    0.071    0.000 {len}
        1    0.071    0.071    0.071    0.071 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.069    0.069    0.103    0.103 getlimits.py:93(__new__)
    38546    0.069    0.000    0.369    0.000 generic.py:750(_consolidate_inplace)
    41900    0.066    0.000    0.568    0.000 index.py:189(dtype)
     5071    0.066    0.000    1.446    0.000 frame.py:2963(take)
    62751    0.063    0.000    0.080    0.000 numeric.py:1810(isscalar)
    26028    0.062    0.000    0.406    0.000 series.py:1210(__iter__)
    30565    0.062    0.000    0.068    0.000 indexing.py:32(__init__)
    47614    0.060    0.000    0.432    0.000 _methods.py:31(_any)
56160/30240    0.059    0.000    3.217    0.000 indexing.py:695(__getitem__)
    82405    0.059    0.000    0.106    0.000 generic.py:651(ndim)
    69979    0.059    0.000    0.268    0.000 index.py:708(get_loc)
    56160    0.056    0.000    1.495    0.000 indexing.py:63(_get_loc)
        2    0.055    0.028    0.055    0.028 index.py:2704(_get_distinct_indexes)
    43080    0.053    0.000    0.114    0.000 series.py:532(__array_finalize__)
    32400    0.051    0.000    0.078    0.000 indexing.py:1043(_maybe_convert_ix)
    85522    0.050    0.000    0.602    0.000 index.py:236(values)
    47614    0.048    0.000    0.480    0.000 {method 'any' of 'numpy.ndarray' objects}
        1    0.047    0.047  115.077  115.077 <string>:1(<module>)
    32508    0.046    0.000    0.457    0.000 _methods.py:27(_prod)
    37582    0.046    0.000    0.046    0.000 {hasattr}
    37795    0.043    0.000    0.232    0.000 generic.py:778(_is_mixed_type)
     5505    0.041    0.000    0.044    0.000 internals.py:1358(_consolidate_check)
    69871    0.040    0.000    0.042    0.000 indexing.py:1068(_is_integer_index)
    26028    0.040    0.000    0.069    0.000 numerictypes.py:735(issubdtype)
        1    0.039    0.039    0.274    0.274 common.py:64(_isnull_new)
    25680    0.039    0.000    0.085    0.000 shape_base.py:8(atleast_1d)
    37795    0.039    0.000    0.083    0.000 internals.py:1363(is_mixed_type)
     5504    0.038    0.000    1.277    0.000 frame.py:386(__init__)
    25920    0.038    0.000    0.064    0.000 series.py:578(_ixs)
     5397    0.038    0.000    0.042    0.000 {sum}
     5077    0.038    0.000    0.039    0.000 index.py:318(__contains__)
    82513    0.037    0.000    0.047    0.000 internals.py:1016(ndim)
     5288    0.037    0.000    0.876    0.000 internals.py:2219(create_block_manager_from_blocks)
    25920    0.037    0.000    2.866    0.000 indexing.py:804(_getitem_tuple)
      109    0.037    0.000    0.037    0.000 {pandas.lib.list_to_object_array}
       40    0.037    0.001    0.037    0.001 {method 'reshape' of 'numpy.ndarray' objects}
    37795    0.036    0.000    0.119    0.000 generic.py:780(<lambda>)
        1    0.035    0.035    0.531    0.531 nanops.py:31(_f)
    43084    0.035    0.000    0.050    0.000 internals.py:1640(_consolidate_inplace)
     5071    0.035    0.000    0.035    0.000 {method 'take' of 'numpy.ndarray' objects}
    13482    0.033    0.000    0.097    0.000 frame.py:749(iterrows)
    43192    0.033    0.000    0.049    0.000 index.py:135(__array_finalize__)
    37471    0.032    0.000    0.046    0.000 indexing.py:1063(_is_integer_dtype)
     5397    0.032    0.000    0.188    0.000 internals.py:989(__init__)
        1    0.031    0.031    0.236    0.236 controlid.py:409(form_control_vectors)
    91527    0.031    0.000    0.031    0.000 {method 'append' of 'list' objects}
    37471    0.030    0.000    0.038    0.000 index.py:879(_get_method)
     6420    0.030    0.000    0.039    0.000 twodim_base.py:170(eye)
      108    0.029    0.000    0.224    0.002 internals.py:1785(insert)
    30240    0.029    0.000    0.107    0.000 series.py:3310(_try_cast)
     5505    0.029    0.000    0.036    0.000 internals.py:1227(shape)
     5071    0.028    0.000    0.223    0.000 index.py:390(take)
        1    0.028    0.028    0.208    0.208 controlid.py:429(form_sensor_vectors)
     4320    0.027    0.000    0.027    0.000 {numpy.core.multiarray.dot}
     5288    0.026    0.000    1.146    0.000 frame.py:530(_init_ndarray)
     5397    0.026    0.000    0.107    0.000 internals.py:1231(_verify_integrity)
    38546    0.026    0.000    0.151    0.000 generic.py:751(<lambda>)
    25920    0.025    0.000    0.025    0.000 {pandas.index.get_value_at}
    81630    0.025    0.000    0.026    0.000 internals.py:1350(is_consolidated)
     6851    0.025    0.000    0.025    0.000 {range}
    25920    0.025    0.000    0.133    0.000 indexing.py:1072(_is_label_like)
     5072    0.024    0.000    0.024    0.000 {numpy.core.multiarray.empty}
    30565    0.023    0.000    0.094    0.000 {setattr}
    12840    0.023    0.000    0.755    0.000 shape_base.py:230(hstack)
    38546    0.021    0.000    0.125    0.000 internals.py:1626(consolidate)
     5071    0.021    0.000    0.035    0.000 common.py:728(_maybe_promote)
      108    0.021    0.000    0.021    0.000 common.py:966(_fill_zeros)
     5072    0.021    0.000    0.027    0.000 generic.py:66(_get_axis_number)
      430    0.020    0.000    0.023    0.000 {method 'all' of 'numpy.ndarray' objects}
        2    0.019    0.010    0.036    0.018 panel.py:1475(_extract_axes_for_slice)
        1    0.019    0.019    0.755    0.755 panel.py:1259(_reduce)
        4    0.018    0.004    0.030    0.007 nanops.py:34(<genexpr>)
     5179    0.017    0.000    3.088    0.001 frame.py:1986(__getitem__)
        4    0.017    0.004    0.017    0.004 {zip}
     5613    0.016    0.000    0.026    0.000 internals.py:37(__init__)
    26004    0.015    0.000    0.036    0.000 numeric.py:462(asanyarray)
    26028    0.014    0.000    0.022    0.000 numerictypes.py:667(issubclass_)
    28405    0.013    0.000    0.013    0.000 controlid.py:89(controls)
        2    0.013    0.006    0.013    0.006 index.py:1341(__new__)
      647    0.012    0.000    0.015    0.000 index.py:293(__iter__)
        1    0.011    0.011    0.314    0.314 nanops.py:207(nanmean)
      108    0.010    0.000    0.011    0.000 expressions.py:54(_evaluate_standard)
     5288    0.010    0.000    0.011    0.000 index.py:269(_engine)
     6916    0.010    0.000    0.010    0.000 controlid.py:120(sensors)
      108    0.009    0.000    0.039    0.000 expressions.py:145(evaluate)
     5071    0.009    0.000    0.013    0.000 common.py:431(_get_take_nd_function)
     5287    0.009    0.000    0.039    0.000 indexing.py:910(_convert_to_index_sliceable)
     5288    0.008    0.000    0.032    0.000 frame.py:5734(_prep_ndarray)
        1    0.007    0.007    0.027    0.027 {method 'max' of 'numpy.ndarray' objects}
    16516    0.006    0.000    0.008    0.000 internals.py:1229(<genexpr>)
    30240    0.006    0.000    0.006    0.000 series.py:521(__init__)
      750    0.005    0.000    0.072    0.000 generic.py:659(__getitem__)
     5288    0.005    0.000    0.022    0.000 index.py:272(<lambda>)
     4321    0.005    0.000    0.056    0.000 index.py:289(is_all_dates)
     5071    0.004    0.000    0.005    0.000 common.py:1525(is_float)
    17595    0.004    0.000    0.004    0.000 internals.py:1177(_get_items)
    10794    0.004    0.000    0.004    0.000 internals.py:1233(<genexpr>)
      327    0.004    0.000    0.004    0.000 {method 'copy' of 'numpy.ndarray' objects}
     5613    0.003    0.000    0.003    0.000 internals.py:76(set_ref_locs)
     4429    0.003    0.000    0.053    0.000 index.py:282(inferred_type)
      430    0.003    0.000    0.029    0.000 numeric.py:2222(array_equal)
      750    0.003    0.000    0.077    0.000 panel.py:365(__getitem__)
     5179    0.003    0.000    0.003    0.000 base.py:45(_constructor)
     6045    0.002    0.000    0.002    0.000 internals.py:128(dtype)
    10142    0.002    0.000    0.002    0.000 {pandas.algos.ensure_int64}
      753    0.002    0.000    0.090    0.000 panel.py:495(iteritems)
      108    0.002    0.000    0.589    0.005 frame.py:3818(_combine_frame)
      324    0.001    0.000    0.002    0.000 shape_base.py:60(atleast_2d)
      108    0.001    0.000    0.239    0.002 internals.py:1722(set)
      108    0.001    0.000    0.230    0.002 frame.py:2183(_sanitize_column)
      108    0.001    0.000    0.003    0.000 internals.py:1904(_add_new_block)
     5077    0.001    0.000    0.001    0.000 {hash}
      108    0.001    0.000    0.110    0.001 index.py:1272(insert)
      755    0.001    0.000    0.030    0.000 index.py:450(equals)
      108    0.001    0.000    0.006    0.000 index.py:347(append)
        1    0.001    0.001    0.001    0.001 {method 'diagonal' of 'numpy.ndarray' objects}
      108    0.001    0.000    0.001    0.000 {operator.sub}
      108    0.001    0.000    0.081    0.001 internals.py:1020(set_axis)
        1    0.001    0.001    0.173    0.173 panel.py:273(_init_dict)
      216    0.001    0.000    0.080    0.000 internals.py:1072(_set_ref_locs)
      108    0.001    0.000    0.240    0.002 generic.py:678(_set_item)
      108    0.001    0.000    0.001    0.000 index.py:378(_ensure_compat_concat)
      108    0.001    0.000    0.025    0.000 frame.py:2499(_align_frame)
      108    0.001    0.000    0.014    0.000 internals.py:2489(_merge_blocks)
      108    0.001    0.000    0.018    0.000 expressions.py:58(_can_use_numexpr)
        1    0.001    0.001    0.001    0.001 internals.py:2368(_stack_arrays)
      108    0.001    0.000    0.060    0.001 frame.py:194(na_op)
      430    0.001    0.000    0.003    0.000 _methods.py:35(_all)
      108    0.000    0.000    0.470    0.004 frame.py:2155(_set_item)
      108    0.000    0.000    0.590    0.005 frame.py:215(f)
        1    0.000    0.000    0.000    0.000 panel.py:503(_get_plane_axes)
      109    0.000    0.000    0.001    0.000 {sorted}
      108    0.000    0.000    0.471    0.004 frame.py:2107(__setitem__)
        1    0.000    0.000    0.000    0.000 {numpy.core.multiarray.putmask}
      324    0.000    0.000    0.005    0.000 index.py:889(reindex)
      108    0.000    0.000    0.030    0.000 expressions.py:82(_evaluate_numexpr)
      108    0.000    0.000    0.226    0.002 common.py:1614(_is_sequence)
      108    0.000    0.000    0.099    0.001 frame.py:2573(reindex)
      216    0.000    0.000    0.001    0.000 frame.py:2717(_reindex_with_indexers)
        2    0.000    0.000    0.058    0.029 panel.py:1539(_extract_axis)
      108    0.000    0.000    0.000    0.000 internals.py:83(set_ref_items)
      108    0.000    0.000    0.004    0.000 internals.py:158(reindex_items_from)
      108    0.000    0.000    0.002    0.000 shape_base.py:179(vstack)
      108    0.000    0.000    0.081    0.001 internals.py:1816(set_items_norename)
      108    0.000    0.000    0.026    0.000 frame.py:2453(align)
      108    0.000    0.000    0.003    0.000 frame.py:2681(_reindex_multi)
      108    0.000    0.000    0.000    0.000 generic.py:675(_clear_item_cache)
      108    0.000    0.000    0.061    0.001 frame.py:3822(_arith_op)
      108    0.000    0.000    0.002    0.000 internals.py:2521(_vstack)
      216    0.000    0.000    0.000    0.000 internals.py:2475(<lambda>)
      216    0.000    0.000    0.000    0.000 internals.py:2479(<lambda>)
        1    0.000    0.000    0.001    0.001 internals.py:2254(form_blocks)
        1    0.000    0.000    0.173    0.173 panel.py:236(_init_data)
      108    0.000    0.000    0.000    0.000 internals.py:2372(_asarray_compat)
      216    0.000    0.000    0.000    0.000 frame.py:466(_init_mgr)
        1    0.000    0.000    0.002    0.002 index.py:2708(_union_indexes)
      218    0.000    0.000    0.000    0.000 index.py:2705(<genexpr>)
      108    0.000    0.000    0.000    0.000 internals.py:2513(_block_shape)
        1    0.000    0.000    0.020    0.020 _methods.py:15(_amax)
        1    0.000    0.000    0.006    0.006 nanops.py:447(_get_counts)
        1    0.000    0.000    0.000    0.000 {pandas.lib.ismember}
        1    0.000    0.000    0.001    0.001 internals.py:2351(_multi_blockify)
        4    0.000    0.000    0.296    0.074 _methods.py:23(_sum)
      109    0.000    0.000    0.030    0.000 {any}
      108    0.000    0.000    0.000    0.000 {method 'copy' of 'dict' objects}
        5    0.000    0.000    0.000    0.000 {min}
        3    0.000    0.000    0.012    0.004 nanops.py:26(check)
        1    0.000    0.000    0.276    0.276 nanops.py:125(_get_values)
        1    0.000    0.000    0.314    0.314 nanops.py:55(f)
        1    0.000    0.000    0.000    0.000 index.py:2749(_sanitize_and_check)
        3    0.000    0.000    0.000    0.000 linalg.py:106(_makearray)
        3    0.000    0.000    0.000    0.000 {max}
        2    0.000    0.000    0.122    0.061 fromnumeric.py:1281(ravel)
      108    0.000    0.000    0.000    0.000 {method 'clear' of 'dict' objects}
        5    0.000    0.000    0.000    0.000 linalg.py:124(_realType)
        1    0.000    0.000    0.000    0.000 index.py:860(isin)
        2    0.000    0.000    0.000    0.000 linalg.py:139(_commonType)
        1    0.000    0.000    0.001    0.001 twodim_base.py:221(diag)
        1    0.000    0.000    0.023    0.023 panel.py:572(_get_values)
        1    0.000    0.000    0.531    0.531 panel.py:1262(<lambda>)
      108    0.000    0.000    0.000    0.000 internals.py:2355(<lambda>)
        1    0.000    0.000    0.755    0.755 panel.py:1642(mean)
      216    0.000    0.000    0.000    0.000 {id}
        1    0.000    0.000    0.000    0.000 nanops.py:543(_ensure_numeric)
        1    0.000    0.000    0.001    0.001 internals.py:2236(create_block_manager_from_arrays)
        4    0.000    0.000    0.000    0.000 {method 'values' of 'dict' objects}
        2    0.000    0.000    0.021    0.011 {method 'sum' of 'numpy.ndarray' objects}
        1    0.000    0.000    0.000    0.000 base.py:702(isspmatrix)
        1    0.000    0.000    0.000    0.000 nanops.py:161(_na_ok_dtype)
        2    0.000    0.000    0.057    0.029 index.py:2689(_get_combined_index)
        6    0.000    0.000    0.000    0.000 linalg.py:111(isComplexType)
        1    0.000    0.000    0.000    0.000 linalg.py:101(get_linalg_error_extobj)
        1    0.000    0.000    0.000    0.000 nanops.py:164(_view_if_needed)
        1    0.000    0.000    0.173    0.173 panel.py:230(__init__)
        1    0.000    0.000    0.000    0.000 {method 'keys' of 'dict' objects}
        1    0.000    0.000    0.001    0.001 panel.py:316(_init_arrays)
        2    0.000    0.000    0.000    0.000 controlid.py:129(validation_data)
        2    0.000    0.000    0.000    0.000 controlid.py:108(gain_omission_matrix)
        1    0.000    0.000    0.000    0.000 linalg.py:219(_assertNoEmpty2d)
        1    0.000    0.000    0.000    0.000 linalg.py:192(_assertRank2)
        1    0.000    0.000    0.000    0.000 common.py:1292(_try_sort)
        1    0.000    0.000    0.000    0.000 {method 'pop' of 'list' objects}
        1    0.000    0.000    0.274    0.274 common.py:47(isnull)
        2    0.000    0.000    0.000    0.000 {method '__array_prepare__' of 'numpy.ndarray' objects}
        1    0.000    0.000    0.000    0.000 internals.py:2379(_shape_compat)
        1    0.000    0.000    0.000    0.000 nanops.py:103(_get_fill_value)
        1    0.000    0.000    0.000    0.000 linalg.py:198(_assertRankAtLeast2)
        1    0.000    0.000    0.000    0.000 index.py:876(_array_values)
        1    0.000    0.000    0.000    0.000 controlid.py:112(gain_omission_matrix)
        1    0.000    0.000    0.000    0.000 {method 'extend' of 'list' objects}
        1    0.000    0.000    0.000    0.000 linalg.py:130(_linalgRealType)
        1    0.000    0.000    0.000    0.000 {method 'iteritems' of 'dict' objects}
        2    0.000    0.000    0.000    0.000 controlid.py:98(identification_data)

@moorepants
Copy link
Member Author

Temporary solution is in place with PR #88.

Also see moorepants/DynamicistToolKit#19 as the problem lies in DynamicistToolkit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@moorepants