Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PERF: Rely on C-level str conversions in loadtxt for up to 2x speedup #19687

Closed
wants to merge 4 commits into from
Closed

PERF: Rely on C-level str conversions in loadtxt for up to 2x speedup #19687

wants to merge 4 commits into from

Commits on Aug 26, 2021

  1. BUG: fix string truncation bug in loadtxt 

    Closes numpy#17277. If loadtxt is passed an unsized string or byte dtype,
the size is set automatically from the longest entry in the first
50000 lines. If longer entries appeared later, they were silently
truncated.
    @DFEvans @anntzer
    DFEvans authored and anntzer committed Aug 26, 2021
    Configuration menu
    Copy the full SHA
    0c33cfd View commit details
    Browse the repository at this point in the history

  2. PERF: In loadtxt, load rows as flat structured dtypes. 

    This is much faster (~30%) for loading actual structured dtypes (by
skipping the recursive packer), somewhat faster (~5-10%) for large loads
(>10000 rows, perhaps because shape inference of the final array is
faster?), and much slower (nearly 2x) for very small loads (10 rows) or
for reads using `dtype=object` (due to the extraneous limitation on
object views, which could be fixed separately); however, the main point
is to allow further optimizations.
    @anntzer
    anntzer committed Aug 26, 2021
    Configuration menu
    Copy the full SHA
    abb705e View commit details
    Browse the repository at this point in the history

  3. PERF: In loadtxt, rely on implicit string conversion. 

    This patch takes advantage of the possibility of assigning a tuple of
*strs* to a structured dtype with e.g. float fields, and have the strs
be implicitly converted to floats by numpy at the C-level.  (A
Python-level fallback is kept to support e.g. hex floats.)  Together
with the previous commit, this provides a massive speedup (~2x on the
loadtxt_dtypes_csv benchmark for 10_000+ ints or floats), but is
beneficial with as little as 100 rows.  Very small reads (10 rows) are
still slower (nearly 2x for object), as well as reads using object
dtypes (due to the extra copy), but the tradeoff seems worthwhile.
    @anntzer
    anntzer committed Aug 26, 2021
    Configuration menu
    Copy the full SHA
    65fee8e View commit details
    Browse the repository at this point in the history

  4. PERF: Implicit check for field count in loadtxt. 

    In the fast-path of loadtxt, the conversion to np.void implicitly checks
the number of fields.  Removing the explicit length check saves ~5% for
the largest loads (100_000 rows) of numeric scalar types.
    @anntzer
    anntzer committed Aug 26, 2021
    Configuration menu
    Copy the full SHA
    a126896 View commit details
    Browse the repository at this point in the history