[BUG] handle } in line delimited json

[joshowen]

• closes #14390
• tests added / passed
• passes git diff upstream/master | flake8 --diff
• whatsnew entry

[codecov-io]

Current coverage is 85.26% (diff: 100%)

Merging #14391 into master will increase coverage by <.01%

@@             master     #14391   diff @@
==========================================
  Files           140        140          
  Lines         50634      50639     +5   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43173      43178     +5   
  Misses         7461       7461          
  Partials          0          0          

Powered by Codecov. Last update d98e982...edb1488

[jreback]

can you add a benchmark for lines=True in asv_bench/benchmarks/packers.py (alongside the other json ones)

[joshowen]

Not sure why that test failed,, the only change was a line of whitespace

[jreback]

looks like some things got repeated here

[joshowen]

Looks like self.N/self.C are repeated in most of these classes. Want me to clean them all up?

[jreback]

sure that would be great (you can also make a common base class(s) if that helps as well)

[jorisvandenbossche]

@joshowen you can leave it here as is. I cleaned this up in another PR (@jreback yes I know, I should merge that ...)

[jreback]

ok that's fine (though @joshowen make sure your example doesn't have dups as this is new code)

[jorisvandenbossche]

ah, yes, it is of course OK to remove the lines in this added code that you do not need for this benchmark

[jreback]

can you also round trip it and user assert_frame_equal on the result (in addition to the above test)

[jreback]

@joshowen can you also post a run for this benchmark (versus previous); can also do it in a %timeit as well. Just checking if any perf issues.

[joshowen]

@jreback is there an easy way to do that? Or should I port the asv test to master and run/compare?

[jreback]

you can run asv if you want, otherwise just do it in ipython (before and after), e.g. something like

In [6]: df = DataFrame(dict([('float{0}'.format(i), np.random.randn(N)) for i in range(C)]), index=index).reset_index(drop=True)

In [7]: df.to_json('foo.json',orient='records',lines=True)

In [8]: %timeit df.to_json('foo.json',orient='records',lines=True)
1 loop, best of 3: 3.66 s per loop

In [9]: %timeit df.to_json('foo.json',orient='records')
10 loops, best of 3: 98.8 ms per loop

and looking at this, we have a BIG perf hit when lines=True (this is a separate issue).

[jreback]

cc @aterrel

[jreback]

@joshowen can you open another issue about the perf

In [15]: %prun df.to_json('foo.json',orient='records',lines=True)
         100058 function calls (100056 primitive calls) in 3.759 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    2.017    2.017    3.656    3.656 json.py:600(_convert_to_line_delimits)
        1    1.002    1.002    1.002    1.002 {method 'join' of 'str' objects}
        1    0.630    0.630    0.630    0.630 {numpy.core.multiarray.array}
        1    0.078    0.078    0.078    0.078 {pandas.json.dumps}
        1    0.012    0.012    0.012    0.012 {method 'write' of 'file' objects}

something odd going on here

[jreback]

lgtm. @jorisvandenbossche

[jorisvandenbossche]

Thanks @joshowen !