OMP Error on Series experssions

[davidljung]

from SO: http://stackoverflow.com/questions/23537265/inconsistent-system-error-with-pandas-0-13-series-operation/23542455#23542455

When performing some expressions involving Series returned from DataFrame column indexing (Pandas 0.13.1 with Numpy 1.8.0 in Windows 7), a new pythonw.exe console window appears (empty) but the initiating GUI indicates the following error text was returned:

OMP: Error #134: Cannot set thread affinity mask.
OMP: System error #87: The parameter is incorrect.

It occurs with expressions such as:
df = df.loc[(df['timediff'] > -20) & (df['timediff'] < 20),:]
and
df['GP'] = (df['Price'] - df['Cost'] - (df['Discount'] / df['Qty'])) * df['Qty']

(this is on moderately small DataFrames < 10 million rows)

As a workaround, disabling numexpr operations seems to avoid it:

pandas.computation.expressions.set_use_numexpr(False)

as suggested in issue #3434.

Possibly related (at least by error text):
http://software.intel.com/en-us/forums/topic/279406

[jreback]

pls show the actual frame; this is a timedelta issue I am pretty sure

[davidljung]

The columns in question are all float64. It didn't make any difference if the comparisons to the float64 timediff column used python float literals rather than ints either.

[jreback]

@davidljung with a reproducible example of exactly what causes the error, it might be possible to see where it is actually coming from.

[jreback]

pls show your Numexpr version

[davidljung]

How/where do I find the "Numexpr" version?
According to the package manager I'm using (Canopy SciPy environment under Windows), I have a package named numexpr 2.2.2-2 installed, which may be it. (not entirely sure what Numexpr is. I took a look in the source of the pandas.computation.expressions module, but don't see any version defined there.)

Unfortunately, I can't provide a reproducible example. I've tried a couple times, but the issue is intermittent. In addition, it seems to only occur for large DataFrames, so whenever I reduced the data size to create a reproduction the issue stops occurring. The full data-set is too big to share and is proprietary anyway.

@jreback: The issue may well be the same one you reported in #3434, so if you can reproduce that it may provide a starting point.

The DataFrame was constructed by reading from a HDFStore and performing a couple selections like this (renamed the table & columns below for brevity):

        df = store.select('Table','ID== %d' % id)
        df = df.loc[~df.UPC.isin(upcs1) & ~df.UPC.isin(upcs2),:]
        df = df.loc[~df.Ticket.isin(ticketIDs),:]

followed by the expression like:

        df['GP'] = (df['Price'] - df['Cost'] - (df['Discount'] / df['Qty'])) * df['Qty']

Note that the HDF file containing the table was not created using HDFStore, but directly via pyTables using the open_file() method, followed by create_table() and repeated calls to table.row.append() to insert the data rows as they were read peace-meal from an ODBC query cursor.

The pyTables description was constructed using the PyTable types IntCol32(), Int64Col(), Float64Col(), StringCol() and one Time64Col() column, though that column has not been referenced in the code or expressions above that cause the error.

[jreback]

import numexpr
numexpr.__version__

what is your numpy/pandas version as well?

try writing the file out using HDFStore, then reading it in again and see if your can reproduce.
(and Time64Col are not supported natively in pandas; they are instead supported via an Int64 Column which as a time-stamp).

And where exactly does the error show up?

The error I mentioned in #3434 I could not reproduce.

[davidljung]

The code above yields version '2.2.2' for numexpr. Pandas & Numpy versions in original comment above.

I don't know how to write the file out using HDFStore - the HDF file is 130GB, so it won't fit into memory (I only have 32GB RAM). If there is a way to 'stream' it through I'd like to know about it so that I don't have to use the PyTables API directly. I've been using
HDFStore.put(table, df, format='table', data_columns=True)
to write out smaller DataFrames that do fit into memory.

I just noticed this method has an append parameter, so perhaps I could read the rows in chunks from my ODBC cursor into a DataFrame in memory and then call that repeatedly with append=True ?

The error occurs on the line above starting with df['GP'] =. I'm using an IPython console that is built into the Canopy UI window. When the error is triggered a new 'windows command'-like window (i.e. black background) opens which is blank and there is no return from the ipython console. Closing the window using the close button results in ipython reporting the process terminated with that error. From memory I tried it with the Qt-based ipython UI window and saw the same error.

[jreback]

their is extensive documentation on using HDF: http://pandas.pydata.org/pandas-docs/stable/io.html#hdf5-pytables. pls post df.info() and df.dtypes

[jreback]

its possible their is something buggy under canopy with their numpy build. you can try installing a separate environment using conda to see if it fixes the problem.

[davidljung]

I selected out a subset of rows. Here is df.info():

<class 'pandas.core.frame.DataFrame'>
Int64Index: 1570804 entries, 0 to 23462204
Data columns (total 24 columns):
Amount                 1570804 non-null float64
Cost                   1570804 non-null float64
DimBusinessDateID      1570804 non-null int32
DimCashierID           1570804 non-null int32
DimItemAccountID       1570804 non-null int32
DimOrgID               1570804 non-null int32
DimPriceBookID         1570804 non-null int32
DimPriceBookPromoID    1570804 non-null int32
DimPromoID             1570804 non-null int32
DimSaleDateID          1570804 non-null int32
DimTillID              1570804 non-null int32
Discount               1570804 non-null float64
FctItemID              1570804 non-null int32
LineID                 1570804 non-null int32
Price                  1570804 non-null float64
Qty                    1570804 non-null float64
Quantity               1570804 non-null int32
SourceStatus           1570804 non-null object
Tax                    1570804 non-null float64
TaxableAmount          0 non-null float64
TicketInternalKey      1570804 non-null int64
UpdateDateTime         1570804 non-null float64
UpdateMethod           1570804 non-null object
Weight                 1570804 non-null object
dtypes: float64(8), int32(12), int64(1), object(3)

and df.dtypes:

Amount                 float64
Cost                   float64
DimBusinessDateID        int32
DimCashierID             int32
DimItemAccountID         int32
DimOrgID                 int32
DimPriceBookID           int32
DimPriceBookPromoID      int32
DimPromoID               int32
DimSaleDateID            int32
DimTillID                int32
Discount               float64
FctItemID                int32
LineID                   int32
Price                  float64
Qty                    float64
Quantity                 int32
SourceStatus            object
Tax                    float64
TaxableAmount          float64
TicketInternalKey        int64
UpdateDateTime         float64
UpdateMethod            object
Weight                  object
dtype: object

I'll report back if I find time to try a separate environment.
Thanks.

[JohnNapier]

Please find here (#9328) code to reproduce this issue.

[jreback]

closing as related to the Enthought distro