BUG/PERF: Stata value labels

[kshedden]

closes #12014

This PR fixes a minor bug and introduces some performance enhancements, all related to value label reading in Stata files.

The bug is that when reading a Stata file incrementally, the value labels will be read even when specifying convert_categoricals=False (this does not happen when reading the entire file at once).

The performance enhancements are:

1. Read key/value information as an ndarray using np.frombuffer rather than as a Python loop.
2. When splitting the value labels at the offsets, we currently pass txt[off[i]:] to null_terminate, which creates many copies of large segments of a potentially large byte array. I changed it so that
   only the relevant part of the string is copied.

Relating to 2, further performance improvements might be possible since there is no trailing null byte to remove except for the last element of txt (thus some of the work in _null_terminate is superfluous).

Background: This is an issue when processing large Stata files with millions of distinct value labels.

[jreback]

ok, sounds gr8!.

just confirm that we have reasonable benchmarks in asv (the packers.py) file is the only one for stata.

[kshedden]

Is there a place to put data files for ASV to read in? The stata writer is more limited than the stata reader, so it's hard to test some features like strls that are not part of dta version 114.

[jreback]

you can just make a directory under the asv_benchmarks

[jreback]

@kshedden want to update

[kshedden]

I added a test but haven't been able to get asv to run it. I don't have conda so switched the asv conf file to use virtualenv. It fails when using pip to install pytables:

(py3)-bash-4.1$ pip install --upgrade pytables Collecting pytables Could not find a version that satisfies the requirement pytables (from versions: ) No matching distribution found for pytables

I'm using python 3.4.2rc1 (and also switched asv conf to use python 3.4)

[jreback]

asv can work with pip but needs a slighty modified config file (e.g. a different one) as pip uses tables (not that conda accepts both of these).

[jreback]

pls add a whatsnew note (in performance). you don't necessarily need to include an asv benchmark (though nice if its easy), but post a perf comparison at the top of the issue.

[pv]

Re conda vs. virtualenv, this may be of interest: spacetelescope/asv#322 (comment) spacetelescope/asv#329

[jreback]

@pv that's a nice feature....care to do a PR to upgrade pandas/asv.conf.json

[jreback]

@kshedden can you update

[jreback]

@kshedden can you update

[kshedden]

Sorry for the delay... I wasn't able to test this in ASV. I updated whatsnew, not sure what else needs to be done.

[jreback]

add this issue number here

[jreback]

looks good
can u run flake8 on the diff and fix any issue

[jreback]

this is also a perf enhancement, yes? pls add a line in Performance (use this PR number for it)

[jreback]

minor comments. pls squash. ping when green.

[kshedden]

@jreback should be good to go

[jreback]

merged via 449ab6b

thanks!