Update EditEngine code to use 32 bit paragraph storage

[DamjanJovanovic]

Our EditEngine, in main/editeng, has a number of containers for paragraphs, paragraph portions, lines, etc. These are based on svl's "PTRARR" type classes, which is just an array of some caller-defined class that grows as needed, like ::std::vector but hardcoded to 16 bit limits. Worse, a lot of code, even outside of main/editeng, is written under these assumptions, and passes 16 bit types and constants to EditEngine classes and retrieves 16 bit results.

One particularly notorious bug emerging from this design, is that when you try to paste an HTML table of more than 65534 cells into Calc, the first 65534 cells are pasted, but all further cells are silently lost, something reported in at least 3 bugs in the last 18 years (57176, 110486 and 117225). This apparently happens because our HTML import is similar to XML DOM parsing, done in 2 phases, first parsing the results into memory, then processing those in-memory results to populate data into the spreadsheet (ScHTMLImport::WriteToDocument() in main/sc/source/filter/html/htmlimp.cxx). When parsing, each HTML cell () becomes a new EditEngine paragraph, and after 65534 paragraphs, all further cells are ignored, so when it's time to populate the spreadsheet, the data isn't there to add.

This enormous patchset changes the container for paragraphs to a new BaseList class, which wraps ::std::vector and uses 32 bit integers to access it. All EditEngine methods are changed to take 32 bit paragraph index parameters, all EditEngine classes store any paragraph indices as 32 bit fields, all 16 bit paragraph index constants like 0xffff are changed to 32 bit 0xffffffff (and to a more readable constant like EE_PARA_NOT_FOUND), and all (known) calling code everywhere in OpenOffice is updated to take these changes into account.

The bug is definitely fixed, all sample documents from Bugzilla paste fully and correctly. What is harder to prove is that nothing else broke. Much had to change, 136 files in 9 modules. OpenGrok helped tremendously in finding obscure places where EditEngine methods were getting called from. I've really tried to avoid introducing any new bugs, but it is hard to be sure with a change of this size. I am making a PR instead of directly pushing to allow you to test a lot ;).

[Pilot-Pirx]

Hi Damjan,
this is a great amount of work, thank you!
I will try to do a build on Windows and test some files...

[cbmarcum]

H Damjan,
Thanks for your work on this.
I made a slight change to the html test file from i117635 by removing the first row so that cell A1 was in the first row.
See link [1].
the file is a simple html table with 20000 rows and 5 columns.
When I open it with my build of this PR on CentOS 7 it still only opens into a Writer/HTML document to the row 12801 before starting to just produce text beyond that. Same as with trunk.

[1] https://home.apache.org/~cmarcum/test-files/20000-row-html-table.html

[DamjanJovanovic]

When I open it with my build of this PR on CentOS 7 it still only opens into a Writer/HTML document to the row 12801 before starting to just produce text beyond that. Same as with trunk.

[1] https://home.apache.org/~cmarcum/test-files/20000-row-html-table.html

Nice catch. Yes, apparently there is still a problem somewhere in Writer. I only tested copying from a web browser and pasting to Calc.

[DamjanJovanovic]

I believe the Writer table limit of 64K cells is unrelated to this issue.

For example in main/sw/source/filter/html/htmltab.cxx:

class HTMLTable
{
...
    sal_uInt16 nRows;                   // Anzahl Zeilen
    sal_uInt16 nCols;                   // Anzahl Spalten

That's not EditEngine related, that's a separate issue specific to Writer.

Please test only copying from a web browser and pasting into Calc, and opening through the "File type" of "HTML document (OpenOffice Calc) (*html, *.htm)".

[cbmarcum]

My latest testing:
File > Open
set file type filter to HTML Document (OpenOffice Calc) (.html;.htm)

trunk got to row 13106
with PR 164 got all 20000 rows

Copy/Paste from Firefox to Calc
trunk and this PR
all 20000 rows for both

Copy/Paste from Firefox to Writer
trunk and this PR
all 20000 rows as tab separated values per row.

[DamjanJovanovic]

Copy/Paste from Firefox to Calc trunk and this PR all 20000 rows for both

There is no way all 20000 pasted successfully over HTML on trunk. Either the transfer format wasn't HTML, or you tested something other than trunk.

[cbmarcum]

Copy/Paste from Firefox to Calc trunk and this PR all 20000 rows for both

There is no way all 20000 pasted successfully over HTML on trunk. Either the transfer format wasn't HTML, or you tested something other than trunk.

Hi Damjan,
You are correct. I found out what was wrong with my test.
I had copied from a Firefox outside of the VM and pasted to AOO inside the VM. This opened a Text Import dialog and did import all 20000 rows in both trunk and the PR.

I can confirm that copying from Firefox and pasting into AOO within the same VM results in:
trunk getting to row 13106 and inserting a lot (or all) of the rest into column D of the next row.
The PR build takes all 20000 rows.
Both without using the Text import dialog.
Sorry for the confusion.

[cbmarcum]

my test was successful

[DamjanJovanovic]

Is everyone happy? Should we merge this to trunk? And 420?

[Pilot-Pirx]

Sorry, I forgot to build this one...

My Windows build is now running, looks good so far.
I will upload it to DropBox later, if anyone is interested in testing it.

Here it is:
https://www.dropbox.com/s/tmf97odgftuej67/Apache_OpenOffice_4.5.0_Win_x86_install_en-US-PR164.exe?dl=0

[Pilot-Pirx]

My test on Windows with this table:
https://home.apache.org/~cmarcum/test-files/20000-row-html-table.html

was successful!

[Pilot-Pirx]

I am having problems with an xlsx file containing Cyrillic characters. Only the numbers are imported. It may only be my machine.

Update:
The file was from https://bz.apache.org/ooo/show_bug.cgi?id=126720 and the fixing commit is most likely not in this branch...