ENH: Data formatting with unicode length

[sinhrks]

Closes #2612. Added display.unicode.east_asian_width options, which calculate text width considering East Asian Width. Enabling this option affects to a performance as width must be calculated per characters.

Current results (captured)

• Basic impl and test
• Series / DataFrame truncation
• Perf test
• Doc / Release note

[shoyer]

How much slower is this than the default behavior? My guess is that anyone who uses these characters will want this.

Usually we're not printing enough data to the screen for performance on this sort of stuff to matter.

[kawochen]

typo - calculate

[kawochen]

east_asian_len calculation can be reduced to a 3-element set membership testing. Ever so slightly faster but probably too micro to matter. I think ambiguous characters need special handling (?).

[sinhrks]

OK, this PR should work all cases which I'm aware of. Appreciated if anyone provide further test cases if any concerns.

@shoyer Yes, east-asian prefer this to be default True. But it is almost 2 times slower in below case.

• DataFrame contains 10000 data, 100 rows * 100 columns, each item contains 10 Unicode chars
• Display options are default:
  • pd.options.display.max_rows: 60
  • pd.options.display.max_columns: 20

import numpy as np
import pandas as pd

chars = list(u'あいうえおかきくけこさしすせそたちつてとなにぬねの')

def rand_jp(x):
    return ''.join(np.random.choice(chars) for _ in range(x))

df = pd.DataFrame(np.empty((100, 100)))
df = df.applymap(lambda x: rand_jp(10))

%timeit unicode(df)
# 10 loops, best of 3: 177 ms per loop

# Enable Unicode handling
pd.options.display.unicode.east_asian_width = True

%timeit unicode(df)
# 1 loops, best of 3: 381 ms per loop

The affect is almost the same as all ascii (same condition except for characters are all ascii):

• Default: 10 loops, best of 3: 131 ms per loop
• Enable Unicode handling: 1 loops, best of 3: 302 ms per loop

@kawochen I may not properly understand, but reducing East Asian Width category will not affect to the performance because dict lookup is O(1).

I think ambiguous characters need special handling (?).

Do you have any idea about affected characters and special handling logic? My concern is these characters can not be aligned properly even if we tried so.

CC: @ayapi

[shoyer]

If I understand correctly, because pandas does not actually print every element of large dataframes, so printing a larger DataFrame would be the same speed?

How does this patch effect the speed of printing Unicode text if it only contains ASCII characters?

[sinhrks]

@shoyer Correct. Larger data can be printed almost the same speed. Perf is not affected by data size once it exceeds max_columns and max_rows.

Result for all ascii are described in above. Almost 2 times longer, because the logic is the same. It can be short-passed if we can distinguish whether input has 2 bytes char or not including symbols. Is it possible using any built-in?.

[shoyer]

I would probably try tweaking the string length check and seeing if a simple variant can give you a speed up. But generally this is pretty reasonable already. Few dataframes will show this many strings.

On Sat, Sep 26, 2015 at 8:16 PM, Sinhrks notifications@github.com wrote:

@shoyer Correct. Larger data can be printed almost the same speed. Perf is not affected by data size once it exceeds max_columns and max_rows.

Result for all ascii are described in above. Almost 2 times longer, because the logic is the same. It can be short-passed if we can distinguish whether input has 2 bytes char or not including symbols. Is it possible using any built-in?.

Reply to this email directly or view it on GitHub:
#11102 (comment)

[kawochen]

@sinhrks Yes the optimization I mentioned is so minor I don't know why I brought it up. It makes east_asian_len take about 30~40% less time to run, but I don't think this is where time is spent anyways. Regarding ambiguous characters I am not sure what can be done (perhaps config/options)? Also wanted to mention in passing that printing ⟼ (neutral) would still not print as one would hope.

[jreback]

pass a different pad function if you are in east_asian_len mode, then then you don't need the if/then block that duplicates code

[sinhrks]

@jreback I've refactored a little based on your comments. I've created TextAdjustment class which has a set of east-asian depending function rather than defining separate pad function. Because separate definitions may causes unexpected results in future (e.g. using pad for east-asian with normal len ). If looks OK, I'll squash.

@kawochen OK, I leave current east_asian_len.

Also, please provide the list of ambiguous characters and its width. As long as I understand, these ambiguous character widths are not integral multiple, thus these cannot be aligned by padding with white spaces. I don't think we can fix it because it is unicode spec.

[jreback]

looks nice!

can u run a perf check on relevant benchmarks to assert its about the same as current?

add a release note and can do for 0.17.0

[kawochen]

@sinhrks ambiguous characters are either wide or narrow. see here for list http://unicode.org/reports/tr11-2/

[sinhrks]

@kawochen Ok. Backed to first discussion, it can't be support it without clarifying the logic. I can't find the logic per characters from your link.... Maybe I misunderstood sonething? Pls provide actual code works as your expectation.

[kawochen]

@sinhrks There is no per character logic. All ambiguous characters are narrow on my terminal, but that's just my settings. Users should know whether it should be treated as wide or narrow, which depends on where the characters are being printed, so making it configurable might make sense. I don't think that can be figured out from within Python. You can try printing the characters in the list.

[sinhrks]

Thanks. Maybe I could understand a little. From the link:

When mapping Unicode to East Asian legacy character encodings

• Wide Unicode characters always map to fullwidth characters.
• Narrow (and neutral) Unicode characters always map to halfwidth characters.
• Halfwidth Unicode characters always map to halfwidth characters.
• Ambiguous Unicode characters always map to fullwidth characters.

When mapping Unicode to non-East Asian legacy character encodings

• Wide Unicode characters do not map to non-East Asian legacy character encodings.
• Narrow (and neutral) Unicode characters always map to regular (narrow) characters.
• Halfwidth Unicode characters do not map.
• Ambiguous Unicode characters always map to regular (narrow) characters.

When mapping Unicode to East Asian legacy character encodings: Ambiguous should be handled ad full-width (length=2). It can be covered by the impl added by the PR (display.unicode.east_asian_width=True) .

When mapping Unicode to non-East Asian legacy character encodings: Because full width are not mapped (cannot appear), all characters including ambiguous can be regarded as half-width (length=1). It should be corresponding to the current default display.unicode.east_asian_width=False.

What the situation requires a separate option only for Ambiguous?

[kawochen]

@sinhrks when we are not mapping to legacy encodings. For example in my terminal Chinese characters are twice as wide as ambiguous characters. Whether it should be 1 or 2 depends on where and how the data will be displayed. In the unlikely case when we wanted it to look pretty in Arial then we'd print tabs to align stuff. In mono space fonts it's easier so we can use spaces. In modern terminals ambiguous characters are usually narrow.

[sinhrks]

@kawochen Can you describe with screenshots and code which I can confirm on my terminal?

[kawochen]

In [2]: print('中文\n\u00A1\u00A1ab')
In [4]: east_asian_width('\u00A1')
Out[4]: 'A'

So if I have all of those in a DataFrame, I might have too much padding.

[sinhrks]

Understood, how about the name unicode.ambiguous_as_wide with default False (length=1) if it is popular in current terminal. If specified True, its length is regarded as 2.

[sinhrks]

Updated to add release note and unicode.ambiguous_as_wide. "Ambiguous" characters cannot be aligned properly on Sphinx (Jupyter).

terminal

Latter case cannot be aligned because of the mismatch between terminal and pandas option.

Sphinx/Jupyter

Going to add note to say "This should be aligned properly in terminal which uses monospaced font."

I'll update perf comparison tomorrow.

[jreback]

?

[jreback]

@sinhrks ok some minor doc fixes. merge when ready.

[sinhrks]

Thanks, updated doc and asv result attached. Results looks random.

All benchmarks:

    before     after       ratio
  [5049b5  ] [53ac28  ]
    19.50ms    22.42ms      1.15  frame_methods.frame_repr_tall.time_frame_repr_talld
     1.30ms     1.43ms      1.10  groupby.series_value_counts.time_value_counts_int64
     3.41μs     3.74μs      1.10  indexing.indexing_frame_get_value.time_indexing_frame_get_value

[sinhrks]

@jreback Could you merge this when you prepare RC?

All: I'm willing to fix if anything is pointed out during RC.

[jreback]

@sinhrks (and @kawochen ) this is fantastic. quite a bit of work and lots of tests yeh!

[sinhrks]

Found doc layout can differ by environment. I'll update notes to describe it.

1st looks OK, 2nd NG

1st looks NG, 2nd OK