BUG: DataFrame.to_stata() uses wrong struct formats and crashes in int64

[bashtage]

The relevant code is

self.DTYPE_MAP = \
    dict(
        lzip(range(1, 245), ['a' + str(i) for i in range(1, 245)]) +
        [
            (251, np.int16),
            (252, np.int32),
            (253, np.int64),
            (254, np.float32),
            (255, np.float64)
        ]
    )

and

self.TYPE_MAP = lrange(251) + list('bhlfd')

which maps h to int32 and l to int64.

http://docs.python.org/2/library/struct.html#format-characters

shows that h is 2 bytes and l is 4, and so trying to run

struct.pack('<l',2**40)

produces an error.

The obvious fix is to use

self.TYPE_MAP = lrange(251) + list('blqfd')

but this will probably produce errors on 32-bit platforms.

[jreback]

cc @PKEuS

@bashtage can you show an actual file that produces an error? on what stata version? (and pandas version)

[bashtage]

Trivial example:

issue = pd.DataFrame([11456230400L],columns=['a'])
issue.to_stata('issue.dta')

using 0.13.1 (Anaconda/x64/Windows)

Simple to reproduce using S&P 500 data from Yahoo! Finance - some of the volumes are > 2**32.

[jreback]

ok! thanks for the report

[jreback]

want to do a PR for this?

[bashtage]

A simple fix in PR #6335. Worried that it may break on 32 bit.

[jseabold]

Indeed, this is incorrect. If you look at the statsmodels version of the reader this has been changed in moving it to pandas. The type maps does not include int64 over there.

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/foreign.py#L289

[jreback]

closed via 689d765