REF: Use `Styler` implementation for `DataFrame.to_latex` #47970

attack68 · 2022-08-04T20:38:08Z

After a year of patching up things in #41649, and @jreback merge of #47864 I can finally propose this for review.

Objective

Maintain DataFrame.to_latex with its existing arguments, adding no arguments
Process the rendering via Styler.to_latex
Eliminate the need for LatexFormatter (code removal not part of this PR) and dual pandas code systems.
Redocument and direct users to the Styler implementation for forward development

Outcome

All arguments in DataFrame.to_latex were replicable, with the exception of col_space which has no impact upon latex render and, ~~I personally don't like anyway. col_space is deprecated with warning and test.~~ col_space is removed.
All original tests pass with minor changes to latex formatting and no significant changes to latex render.
Some default formatting of floats changes based on pandas Styler options and DataFrame options crossover, which should be addressed later.
The performance of Styler is marginally better, although for the table sizes that one would like to render in latex is neglible, anyway.

~~No whats_new: awaiting feedback.~~

New docs

The key new section of the docs..

attack68 · 2022-08-08T20:22:19Z

@ivanovmg @rhshadrach you both had input to the underlying issue so I think your opinion on the output here is very welcome.

mroeschke · 2022-08-08T22:18:11Z

pandas/tests/io/test_common.py

@@ -351,6 +350,8 @@ def test_read_fspath_all(self, reader, module, path, datapath):
        ],
    )
    def test_write_fspath_all(self, writer_name, writer_kwargs, module):
+        if writer_name in ["to_latex"]:  # uses Styler implementation
+            pytest.importorskip("jinja2")


So this new implementation will require jinja2 to be installed first?

correct. @jreback was fine for this to be the case.

No objection in having that as a requirement, but we regard this as a breaking change, yes?

it is a breaking chang, yes. Current implemtations of dataframe.to_latex will fail after this PR if the user does not have jinja2.

fallback option: since the pr does not remove the LatexFormatter, code could be redirected on undetected jinja2. this would then be non-breaking.

If the fallback behavior was implemented then I think this PR would be okay for 1.5. If not, this could be one of the "breaking without deprecation" behaviors for 2.0 possibly if this change is "too large" #44823

When thinking about doing this I realised the tests that I have altered will be a nightmare for dual implementation, so if this needs to go to 2.0, I don't oppose that.

But then for 2.0 I will probably look to change the arg signature of all of this anyway.

What would the args be changed to? The current arguments should be deprecated.

pandas/core/generic.py

ivanovmg · 2022-08-09T02:52:57Z

pandas/tests/io/formats/test_format.py

@@ -3377,14 +3376,16 @@ def test_filepath_or_buffer_arg(
    filepath_or_buffer_id,
 ):
    df = DataFrame([data])
+    if method in ["to_latex"]:  # uses styler implementation


Does it make sense to simplify it to method == 'to_latex'?

if other methods move over to Styler, (the intention being to_html) we will need a list here anyway

pandas/tests/io/formats/test_format.py

ivanovmg

Some of my comments so far.

rhshadrach

Need to take a more detailed look, but definitely +1 on the direction this is going. I have concerns on merging this prior to 2.0 though.

pandas/core/generic.py

rhshadrach · 2022-08-09T21:06:22Z

pandas/tests/io/test_common.py

@@ -351,6 +350,8 @@ def test_read_fspath_all(self, reader, module, path, datapath):
        ],
    )
    def test_write_fspath_all(self, writer_name, writer_kwargs, module):
+        if writer_name in ["to_latex"]:  # uses Styler implementation
+            pytest.importorskip("jinja2")


No objection in having that as a requirement, but we regard this as a breaking change, yes?

attack68 · 2022-08-10T07:53:41Z

I have concerns on merging this prior to 2.0 though.

Can you share those. Im abivalent as towards 1.5.0 and 2.0, but here are some good reasons for 1.5.0.

It allows getting some feedback issues ahead of 2.0 for these transition methods. Since to_html is also planned for the same transition, but I imagine that is a much more popular method, this feedback might be useful.
This implementation doesn't change the current arguments for DataFrame.to_latex so provides minimal breaking change and a smoother transition. between 1.4. to 1.5 to 2.

Arguments for including in 2.0

Breaking issues do not matter as much.
The keyword arguments to DataFrame.to_latex can be changed (rather simplified) and make this function much simpler in code. (Importantly they do not have to conform to the current arguments)

github-actions · 2022-09-22T00:07:56Z

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

rhshadrach · 2022-09-24T14:10:20Z

Apologies @attack68 on not getting back to you here.

I have concerns on merging this prior to 2.0 though.

Can you share those. Im abivalent as towards 1.5.0 and 2.0, but here are some good reasons for 1.5.0.

The concerns are the breaking changes this introduces, highlighted in my comments above. I think it would be okay to have the added requirement of jinja2 in 2.0 (#47970 (comment)). #47970 (comment) is still outstanding.

…lement # Conflicts: # pandas/tests/frame/test_repr_info.py

mroeschke · 2023-01-05T23:00:09Z

doc/source/whatsnew/v2.0.0.rst

+.. _whatsnew_200.api_breaking.to_latex:
+
+DataFrame to LaTeX has a new render engine
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


Suggested change

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sphinx allows underline to be exact or longer than the text so this isn't strictly necessary, but committed anyway.

mroeschke · 2023-01-05T23:01:10Z

doc/source/whatsnew/v2.0.0.rst

+
+The pandas options below are no longer used and will be removed in future releases.
+The alternative options giving similar functionality are indicated below:
+  - ``display.latex.escape``: replaced with ``styler.format.escape``,


I think this indentation is causing this docbuild error:

2023-01-05T18:42:10.0685513Z /home/runner/work/pandas/pandas/doc/source/whatsnew/v2.0.0.rst:435: WARNING: Block quote ends without a blank line; unexpected unindent.

lets see if the pushed change fixes that error

mroeschke · 2023-01-07T01:22:30Z

Also as a reminder. Could you remove any potential FutureWarnings that are filtered out in the test suite due to the original deprecation?

mroeschke · 2023-01-12T23:57:48Z

Just noting that during yesterday's dev call that we decided that this PR isn't necessarily a blocker for releasing 2.0 i.e. this FutureWarning will persist until 3.0 if not ready.

That being said this PR seems close

Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>

doc/source/whatsnew/v2.0.0.rst

attack68 · 2023-01-13T07:19:06Z

Just noting that during yesterday's dev call that we decided that this PR isn't necessarily a blocker for releasing 2.0 i.e. this FutureWarning will persist until 3.0 if not ready.

That being said this PR seems close

Yes, if I can just get this to green I dont think there is more to do.

Post PR clean up, such as removal of redundant code and checking the filters can be done after 2.0. WHilst not a blocker I still it would be good to get in if poss.

doc/source/whatsnew/v2.0.0.rst

…lement

attack68 · 2023-01-18T06:21:27Z

@ivanovmg @rhshadrach @mroeschke this is greenish now. I believe the http doc build is unrelated.
Please re look and consider if all your comments are addressed

pandas/core/generic.py

mroeschke · 2023-01-19T20:01:54Z

Thanks for the great work here @attack68!

So to clarify the follow ups?

Removing any remaining, related warning filtering
Removing the display options that are no longer relevant?

attack68 · 2023-01-19T20:11:26Z

Yes and remove the redundant LatexFormatter code

…#47970) * Base implementation * Base implementation * test fix up * test fix up * test fix up * doc change * doc change * doc change * mypy fixes * ivanov doc comment * ivanov doc comment * rhshadrach reduction * change text from 1.5.0 to 2.0.0 * remove argument col_space and add whatsnew * mroeschke requests * mroeschke requests * pylint fix * Whats new text improvements and description added * Update doc/source/whatsnew/v2.0.0.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Update doc/source/whatsnew/v2.0.0.rst * remove trailing whitespace * remove trailing whitespace * Whats new linting fixes * mroeschke requests Co-authored-by: JHM Darbyshire (iMac) <attack68@users.noreply.github.com> Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>

attack68 mentioned this pull request Aug 4, 2022

REF: Transfer DataFrame.to_latex to Styler implementation for v1.5.0 #45382

Closed

4 tasks

attack68 added this to the 1.5 milestone Aug 5, 2022

attack68 added Styler conditional formatting using DataFrame.style IO LaTeX to_latex labels Aug 5, 2022

mroeschke reviewed Aug 8, 2022

View reviewed changes

ivanovmg reviewed Aug 9, 2022

View reviewed changes

pandas/core/generic.py Outdated Show resolved Hide resolved

ivanovmg reviewed Aug 9, 2022

View reviewed changes

pandas/tests/io/formats/test_format.py Outdated Show resolved Hide resolved

ivanovmg reviewed Aug 9, 2022

View reviewed changes

attack68 added 10 commits August 9, 2022 07:10

Base implementation

02d7d0c

Base implementation

0c12f3a

test fix up

d644065

test fix up

9dcd254

test fix up

cd05038

doc change

c625610

doc change

493b884

doc change

41fa426

mypy fixes

2d5c419

ivanov doc comment

c9c61c3

attack68 force-pushed the to_latex_styler_implement branch from 82340d1 to c9c61c3 Compare August 9, 2022 05:15

ivanov doc comment

ab6d3ec

rhshadrach reviewed Aug 9, 2022

View reviewed changes

rhshadrach reduction

b913583

mroeschke modified the milestones: 1.5, 2.0 Aug 22, 2022

github-actions bot added the Stale label Sep 22, 2022

mroeschke mentioned this pull request Jan 5, 2023

DEPR: Enforce 2.0 deprecations #50579

Closed

attack68 added 2 commits January 5, 2023 19:08

Merge remote-tracking branch 'upstream/main' into to_latex_styler_imp…

3c2f964

…lement # Conflicts: # pandas/tests/frame/test_repr_info.py

Whats new text improvements and description added

0b4144a

mroeschke reviewed Jan 5, 2023

View reviewed changes

datapythonista mentioned this pull request Jan 6, 2023

RLS: 2.0 #46776

Closed

1 task

Update doc/source/whatsnew/v2.0.0.rst

fae1b3f

Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>

attack68 commented Jan 13, 2023

View reviewed changes

doc/source/whatsnew/v2.0.0.rst Outdated Show resolved Hide resolved

Update doc/source/whatsnew/v2.0.0.rst

9c7d780

attack68 commented Jan 16, 2023

View reviewed changes

doc/source/whatsnew/v2.0.0.rst Outdated Show resolved Hide resolved

attack68 commented Jan 16, 2023

View reviewed changes

doc/source/whatsnew/v2.0.0.rst Outdated Show resolved Hide resolved

attack68 and others added 4 commits January 16, 2023 12:41

remove trailing whitespace

c0bdc6a

remove trailing whitespace

c740279

Merge remote-tracking branch 'upstream/main' into to_latex_styler_imp…

9b6bba8

…lement

Whats new linting fixes

6b314cd

mroeschke reviewed Jan 18, 2023

View reviewed changes

pandas/core/generic.py Show resolved Hide resolved

mroeschke reviewed Jan 18, 2023

View reviewed changes

pandas/core/generic.py Outdated Show resolved Hide resolved

mroeschke reviewed Jan 18, 2023

View reviewed changes

pandas/core/generic.py Outdated Show resolved Hide resolved

mroeschke requests

d6333cb

mroeschke approved these changes Jan 19, 2023

View reviewed changes

mroeschke merged commit 5e4ea2e into pandas-dev:main Jan 19, 2023

attack68 deleted the to_latex_styler_implement branch January 19, 2023 21:42

attack68 mentioned this pull request Apr 20, 2023

DEPR: LatexFormatter in DataFrame.to_latex in favour of Styler #41649

Closed

17 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

REF: Use `Styler` implementation for `DataFrame.to_latex` #47970

REF: Use `Styler` implementation for `DataFrame.to_latex` #47970

attack68 commented Aug 4, 2022 •

edited

attack68 commented Aug 8, 2022

mroeschke Aug 8, 2022

attack68 Aug 9, 2022

rhshadrach Aug 9, 2022

attack68 Aug 10, 2022

mroeschke Aug 10, 2022

attack68 Aug 11, 2022

rhshadrach Aug 11, 2022

ivanovmg Aug 9, 2022

attack68 Aug 9, 2022

ivanovmg left a comment

rhshadrach left a comment

rhshadrach Aug 9, 2022

attack68 commented Aug 10, 2022

github-actions bot commented Sep 22, 2022

rhshadrach commented Sep 24, 2022 •

edited

mroeschke Jan 5, 2023

attack68 Jan 13, 2023

mroeschke Jan 5, 2023

attack68 Jan 13, 2023

mroeschke commented Jan 7, 2023

mroeschke commented Jan 12, 2023

attack68 commented Jan 13, 2023 •

edited

attack68 commented Jan 18, 2023

mroeschke commented Jan 19, 2023

attack68 commented Jan 19, 2023

	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

REF: Use Styler implementation for DataFrame.to_latex #47970

REF: Use Styler implementation for DataFrame.to_latex #47970

Conversation

attack68 commented Aug 4, 2022 • edited

Objective

Outcome

New docs

attack68 commented Aug 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ivanovmg left a comment

Choose a reason for hiding this comment

rhshadrach left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

attack68 commented Aug 10, 2022

github-actions bot commented Sep 22, 2022

rhshadrach commented Sep 24, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mroeschke commented Jan 7, 2023

mroeschke commented Jan 12, 2023

attack68 commented Jan 13, 2023 • edited

attack68 commented Jan 18, 2023

mroeschke commented Jan 19, 2023

attack68 commented Jan 19, 2023

REF: Use `Styler` implementation for `DataFrame.to_latex` #47970

REF: Use `Styler` implementation for `DataFrame.to_latex` #47970

attack68 commented Aug 4, 2022 •

edited

rhshadrach commented Sep 24, 2022 •

edited

attack68 commented Jan 13, 2023 •

edited