test.pandas() - ValueError: The truth value of a DataFrame is ambiguous

[scubamut]

I'm trying to use ta-lib with pandas, but get error (running pandas.test()):

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

[mrjbq7]

Do you have a traceback I can look at?

[scubamut]

Thks for quick reply!

Here's my code and traceback:
I'm using pandas 0.13.0 and ta_lib 0.4.7

import talib as t, pandas as pd
import traceback
df = pd.read_csv('http://ichart.finance.yahoo.com/table.csv?s=SPY&d=0&e=27&f=2014&g=d&a=0&b=29&c=1993&ignore=.csv')
sma = t.abstract.SMA
try : sma(df, 20)
except : traceback.print_exc(file=sys.stdout)

Traceback (most recent call last):
File "", line 5, in
try : sma(df, 20)
File "abstract.pyx", line 330, in talib.abstract.Function.call (talib\abstract.c:5795)
File "abstract.pyx", line 264, in talib.abstract.Function.set_function_args (talib\abstract.c:4927)
File "abstract.pyx", line 223, in talib.abstract.Function.get_parameters (talib\abstract.c:4157)
File "abstract.pyx", line 374, in talib.abstract.Function.get_opt_input_value (talib\abstract.c:6628)
File "C:\Python27\lib\site-packages\pandas\core\generic.py", line 665, in __nonzero
.format(self.class.name))
ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

[mrjbq7]

Okay, the support for pandas hasn't been released as 0.4.8 yet, was letting it sit in master and get tested by people... but this works fine:

import talib as t, pandas as pd
import traceback
df = pd.read_csv('http://ichart.finance.yahoo.com/table.csv?s=SPY&d=0&e=27&f=2014&g=d&a=0&b=29&c=1993&ignore=.csv')
df.columns = [s.lower() for s in df.columns] # ta-lib looks for "open" not "Open", etc.
sma = t.abstract.SMA
sma(df, 20)

Right now the only problem I saw was that ta-lib is looking for lowercase column names, like "open" and "close", not "Open" and "Close"...

Want to try the latest master (should be very stable) and see what you think?

[mrjbq7]

If it works for you, I can do a 0.4.8 release since I was planning one anyway and this would be nice to check working first.

[scubamut]

I'd like to try the latest master but can't seem to install it (64 bit system) - it stops while building 'talib.common' extension, but no error message : see below. Do I need VS to do a build?

E:\Downloads\ta-lib-master>python setup.py install
running install
running build
running build_py
running build_ext
skipping 'talib\common.c' Cython extension (up-to-date)
building 'talib.common' extension
error:

[mrjbq7]

It should have the same requirements as easy_install.

Should be VS to build c extension, and cython possibly because the setup.py uses it of the C files are out of date.

[mrjbq7]

Looks like there was an error message that didn't paste?

[scubamut]

No, there was no more to paste. But I tried this and got a bit further -

E:\Downloads\ta-lib-master>python setup.py build -c mingw32
running build
running build_py
running build_ext
skipping 'talib\common.c' Cython extension (up-to-date)
building 'talib.common' extension
gcc -mno-cygwin -mdll -O -Wall -IC:\Python27\lib\site-packages\numpy-1.7.1-py2.7
-win32.egg\numpy\core\include -Ic:\ta-lib\c\include -IC:\Python27\include -IC:\P
ython27\PC -c talib\common.c -o build\temp.win32-2.7\Release\talib\common.o
error: command 'gcc' failed: No such file or directory

[mrjbq7]

So you're trying to build with mingw32's gcc? I haven't tested that combination -- is it saying gcc isn't found?

[scubamut]

So now I have the same problem another user had a year ago:

alex314159 commented a year ago

---

ok some progress here - I uninstalled Python 64bit and replaced by the 32bit version and it seems to go a lot further into the process (in particular the file above is no longer a problem) I now get a bunch of these warnings:

C:\Python27\Scripts\gcc.exe -mno-cygwin -shared -s build\temp.win32-2.7\Release\talib.o build\temp.win32-2.7\Release\talib.def -Lc:\ta-lib\c\lib -LC:\Python27\l
ibs -LC:\Python27\PCbuild -lta_libc_cdr -lpython27 -lmsvcr90 -o build\lib.win32-2.7\talib.pyd
Warning: .drectve `/manifestdependency:"type='win32' name='Microsoft.VC80.CRT' version='8.0.50727.762' processorArchitecture='x86' publicKeyToken='1fc8b3b9a1e18
e3b'" /DEFAULTLIB:"MSVCRT" /DEFAULTLIB:"OLDNAMES" ' unrecognized

followed by:

c:\ta-lib\c\lib/ta_libc_cdr.lib(../../../../../temp/cdr/ta_HT_TRENDMODE.obj):(.text[TAHT_TRENDMODE]+0x7eb): undefined reference to _ftol2_sse'
c:\ta-lib\c\lib/ta_libc_cdr.lib(../../../../../temp/cdr/ta_HT_TRENDMODE.obj):(.text[_TA_S_HT_TRENDMODE]+0x7eb): undefined reference toftol2sse'
c:\ta-lib\c\lib/ta_libc_cdr.lib(../../../../../temp/cdr/ta_HT_TRENDLINE.obj):(.text[TAHT_TRENDLINE]+0x784): undefined reference to _ftol2_sse'
c:\ta-lib\c\lib/ta_libc_cdr.lib(../../../../../temp/cdr/ta_HT_TRENDLINE.obj):(.text[_TA_S_HT_TRENDLINE]+0x784): undefined reference toftol2sse'
c:\ta-lib\c\lib/ta_libc_cdr.lib(../../../../../temp/cdr/ta_HT_SINE.obj):(.text[TAHT_SINE]+0x79f): undefined reference to _ftol2_sse'
c:\ta-lib\c\lib/ta_libc_cdr.lib(../../../../../temp/cdr/ta_HT_SINE.obj):(.text[_TA_S_HT_SINE]+0x79f): more undefined references toftol2sse' follow
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

he 'solved' the problem by using the VS2008 C compiler instead of mingw32. Unfortunately I don't have that compiler and it seems it's no longer available - do you possibly know where I could find a copy?

Which compiler did you use?

[mrjbq7]

Hmm, I just used whatever free VS edition is available from MS. I'm curious how easy_install worked for you but "doing it by hand" is having problems.

My could try releasing 0.4.8 and you could try easy_install?

(It's not like the C code that cython generates is complicated, sounds more like a configuration problem on Windows)

[scubamut]

alex314159 solved it by using easy_install - but he had a copy of the vs2008 compiler! I'm still stuck with the same problem he experienced, as above. I think the problem is the -lmsvcr90 in the command line. There are too many unresolved references to `_ftol2_sse'

c:\ta-lib\c\lib/ta_libc_cdr.lib(../../../../../temp/cdr/ta_HT_TRENDLINE.obj):(.t
ext[_TA_HT_TRENDLINE]+0x784): undefined reference to _ftol2_sse' c:\ta-lib\c\lib/ta_libc_cdr.lib(../../../../../temp/cdr/ta_HT_TRENDLINE.obj):(.t ext[_TA_S_HT_TRENDLINE]+0x784): more undefined references to_ftol2_sse' follow
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

Any idea what library -lmsvcr90 is linking to?

[mrjbq7]

No, sorry. However I just released master as 0.4.8 if you want to try easy_install.

[scubamut]

Tried 0.4.8 with easy_install - same error. Guess I'll just wait 'til Christoph Gohlke makes a new binary.... I have a workaround to use pandas in the meantime.

[mrjbq7]

Oh, I didn't know you installed from a binary -- thats interesting. Frustrating that install from source is always so difficult on Windows... sorry!

[davidwynter]

Hi,

I get the same error:

File "/home/david/git/pypredict/create_dataset_classifier.py", line 38, in overlap_studies
midpr14 = MIDPRICE(highs.values , lows.values, 14) #Midpoint Price over period
File "abstract.pyx", line 352, in talib.abstract.Function.call (talib/abstract.c:6220)
File "abstract.pyx", line 276, in talib.abstract.Function.set_function_args (talib/abstract.c:5143)
File "abstract.pyx", line 234, in talib.abstract.Function.get_parameters (talib/abstract.c:4363)
File "abstract.pyx", line 400, in talib.abstract.Function.__get_opt_input_value (talib/abstract.c:7132)
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

I built talib on Xubuntu 64 bit from master about a week ago. I use lowercase column names.

Here is my code:
highs = pd.DataFrame(inputs, columns=['high'])
lows = pd.DataFrame(inputs, columns=['low'])
midpr14 = MIDPRICE(highs.values , lows.values, 14) #Midpoint Price over period

Is this the same issue as scubamut was having?

[mrjbq7]

Hmm, it's possible that this check (https://github.com/mrjbq7/ta-lib/blob/master/talib/abstract.pyx#L400) should be if value is not None, but I think your inputs aren't right either.

If you use a Series instead of a DataFrame, does it work? The shape wouldn't be right, otherwise, no?

[davidwynter]

I tried using Series as follows, forgive my newness to python:

midpr14 = MIDPRICE(to_series(highs) , to_series(lows), 14)  #Midpoint Price over period

using:

def to_series(self, concentrate_labels = None):
    """
    Transform a frame via ravelling it

    Parameters
    ----------
    concentrate_labels if not None provides a string to concentrate the index/column labels
       for each entry

    Returns
    -------
    y : Series
    """

    N, K = self.shape

    # use this as the concentration character
    if concentrate_labels is not None:
        index = []
        for l in self.index:
            index.extend([ "%s%s%s" % (l,concentrate_labels,c) for c in self.columns ])
        index = pd.Index(index)

    else:
         index = pd.Index(np.array(self.index).repeat(K))

    return pd.Series(self.values.ravel(), index=index)

No change to the problem.

[davidwynter]

I tried making the change you suggested,if value is not None: , but for some reason now the "python setup.py install" does not work I get this:

david@david-desktop:~/git/ta-lib$ python setup.py install
running install
running build
running build_py
running build_ext
skipping 'talib/common.c' Cython extension (up-to-date)
skipping 'talib/func.c' Cython extension (up-to-date)
cythoning talib/abstract.pyx to talib/abstract.c
building 'talib.abstract' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include -I/usr/local/include -I/opt/include -I/opt/local/include -I/usr/include/python2.7 -c talib/abstract.c -o build/temp.linux-x86_64-2.7/talib/abstract.o
talib/abstract.c: In function ‘__pyx_pf_5talib_8abstract_14_ta_getFuncInfo’:
talib/abstract.c:8210:19: error: storage size of ‘__pyx_v_retCode’ isn’t known
talib/abstract.c:8230:3: warning: passing argument 2 of ‘TA_GetFuncInfo’ from incompatible pointer type [enabled by default]
/usr/local/include/ta-lib/ta_abstract.h:210:12: note: expected ‘const struct TA_FuncInfo **’ but argument is of type ‘struct TA_FuncInfo **’
talib/abstract.c:8210:19: warning: unused variable ‘__pyx_v_retCode’ [-Wunused-variable]
talib/abstract.c: In function ‘__pyx_pf_5talib_8abstract_16_ta_getInputParameterInfo’:
talib/abstract.c:8524:19: error: storage size of ‘__pyx_v_retCode’ isn’t known
talib/abstract.c:8548:3: warning: passing argument 3 of ‘TA_GetInputParameterInfo’ from incompatible pointer type [enabled by default]
/usr/local/include/ta-lib/ta_abstract.h:440:12: note: expected ‘const struct TA_InputParameterInfo **’ but argument is of type ‘struct TA_InputParameterInfo **’
talib/abstract.c:8524:19: warning: unused variable ‘__pyx_v_retCode’ [-Wunused-variable]
talib/abstract.c: In function ‘__pyx_pf_5talib_8abstract_18_ta_getOptInputParameterInfo’:
talib/abstract.c:8817:19: error: storage size of ‘__pyx_v_retCode’ isn’t known
talib/abstract.c:8840:3: warning: passing argument 3 of ‘TA_GetOptInputParameterInfo’ from incompatible pointer type [enabled by default]
/usr/local/include/ta-lib/ta_abstract.h:444:12: note: expected ‘const struct TA_OptInputParameterInfo **’ but argument is of type ‘struct TA_OptInputParameterInfo **’
talib/abstract.c:8817:19: warning: unused variable ‘__pyx_v_retCode’ [-Wunused-variable]
talib/abstract.c: In function ‘__pyx_pf_5talib_8abstract_20_ta_getOutputParameterInfo’:
talib/abstract.c:9147:19: error: storage size of ‘__pyx_v_retCode’ isn’t known
talib/abstract.c:9173:3: warning: passing argument 3 of ‘TA_GetOutputParameterInfo’ from incompatible pointer type [enabled by default]
/usr/local/include/ta-lib/ta_abstract.h:448:12: note: expected ‘const struct TA_OutputParameterInfo **’ but argument is of type ‘struct TA_OutputParameterInfo **’
talib/abstract.c:9147:19: warning: unused variable ‘__pyx_v_retCode’ [-Wunused-variable]
talib/abstract.c: In function ‘__pyx_f_5talib_8abstract___ta_getFuncHandle’:
talib/abstract.c:10032:3: warning: passing argument 2 of ‘TA_GetFuncHandle’ from incompatible pointer type [enabled by default]
/usr/local/include/ta-lib/ta_abstract.h:154:12: note: expected ‘const TA_FuncHandle **’ but argument is of type ‘TA_FuncHandle **’
talib/abstract.c: In function ‘__pyx_f_5talib_8abstract___ta_paramHolderAlloc’:
talib/abstract.c:10088:19: error: storage size of ‘__pyx_v_retCode’ isn’t known
talib/abstract.c:10088:19: warning: unused variable ‘__pyx_v_retCode’ [-Wunused-variable]
talib/abstract.c: In function ‘__pyx_f_5talib_8abstract___ta_setOptInputParamInteger’:
talib/abstract.c:10238:19: error: storage size of ‘__pyx_v_retCode’ isn’t known
talib/abstract.c:10238:19: warning: unused variable ‘__pyx_v_retCode’ [-Wunused-variable]
talib/abstract.c: In function ‘__pyx_f_5talib_8abstract___ta_setOptInputParamReal’:
talib/abstract.c:10314:19: error: storage size of ‘__pyx_v_retCode’ isn’t known
talib/abstract.c:10314:19: warning: unused variable ‘__pyx_v_retCode’ [-Wunused-variable]
talib/abstract.c: In function ‘__pyx_f_5talib_8abstract___ta_getLookback’:
talib/abstract.c:10391:19: error: storage size of ‘__pyx_v_retCode’ isn’t known
talib/abstract.c:10391:19: warning: unused variable ‘__pyx_v_retCode’ [-Wunused-variable]
talib/abstract.c: At top level:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/__multiarray_api.h:1532:1: warning: ‘_import_array’ defined but not used [-Wunused-function]
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/__ufunc_api.h:226:1: warning: ‘_import_umath’ defined but not used [-Wunused-function]
error: command 'gcc' failed with exit status 1

Ideas?

[mrjbq7]

Haha, github shouldn't close automatically for "try and fix" messages!

Anyway, maybe you can try the current master and see if it fixes your problem?

[davidwynter]

I checked it out but still get these errors
talib/abstract.c:8210:19: error: storage size of ‘__pyx_v_retCode’ isn’t known
When I run python setup install

I ran this successfully before, cannot understand what is different except I installed a bunch of python libs as follows:

472  sudo easy_install deap
475  sudo apt-get install graphviz
477  sudo apt-get install libgraphviz-dev
478  sudo easy_install pygraphviz
479  sudo easy_install scoop
480  sudo easy_install neurolab
481  sudo easy_install pip
482  sudo pip uninstall deap
483  sudo pip install deap==0.9.2
497  sudo easy_install node
498  sudo easy_install nose

Not sure any of these would alter things at all?

[mrjbq7]

Hmm, maybe you should try "make clean" then make, see if it works without errors before trying to install it? It will clean build output and then build in-place.

[davidwynter]

I tried this

511  cd ta-lib
515  sudo make clean
516  sudo make install

It passed fine but I got the same errors when running sudo python setup install

Maybe a gcc update by Xubuntu since I last did it? Seems unlikely

[mrjbq7]

Odd, but if you ran make install, should have installed it for you - does your test case pass?

[davidwynter]

No it does not pass, make install only install ta-lib, the C fn library it seems. This is what I expect, the python setup install then installs the python wrapper which uses the aforementioned C lib.

Maybe if I change the args passed in I can use the original one?

I use this to build inputs
inputs = pd.DataFrame.from_csv(sys.argv[1] , sep=',', index_col=0, parse_dates=True )

Then if I call say BBCODE like this
bbupper, bbmiddle, bblower = BBANDS(inputs, 20, 2, 2)
It passes (although bbupper etc are strings, not what I expected?)
But like this
bbupper, bbmiddle, bblower = BBANDS(inputs.values, 20, 2, 2)
It gets the error

Here is the example using MIDPRICE
highs = inputs['high']

File "/home/david/git/pypredict/create_dataset_classifier.py", line 38, in overlap_studies
  midpr14 = MIDPRICE(highs , lows, 14)          #Midpoint Price over period
File "abstract.pyx", line 352, in talib.abstract.Function.__call__ (talib/abstract.c:6220)
File "abstract.pyx", line 276, in talib.abstract.Function.set_function_args (talib/abstract.c:5143)
File "abstract.pyx", line 234, in talib.abstract.Function.get_parameters (talib/abstract.c:4363)
File "abstract.pyx", line 400, in talib.abstract.Function.__get_opt_input_value (talib/abstract.c:7132)
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

My understanding is that inputs.values converts from a DataFrame to an ndarray, which according to the README.me that is what is expected.

[fanbin]

Hi, is there any reason that talib does not accept "Close" but "close"? It could be easily interchanged internally anyway.

[mrjbq7]

Hi @fanbin, we just defaulted to lowercase column names, it is probably not too difficult to change this if it is needed. Are you looking to use titlecase (e.g., Close and Open) as the column names in the data frame / dict that is passed in?

[aking1012]

If it's so easy to implement at core, how is it so hard to implement as a layer around core. Just a little food for thought. Sort of an "If core doesn't want bloat, and it's easy to do - why not only run that branch when you definitely need to run it" sort of thing.