[bug-57342] Excel compatible Zip64 implementation

[rzymek]

https://bz.apache.org/bugzilla/show_bug.cgi?id=57342

I did an in depth analysis of this issue. Turns out the problem is not with the OOXML data generated by POI. The problem has to do with the ZIP format. Specifically with ZIP64 extension. That's why it's all OK up until sheet1.xml reaches over 4GB (uncompressed).
I have all the details written up in a blog post: https://rzymek.github.io/post/excel-zip64/
Short story: Excel will want to repair the file if uncompressed size of a zip entry exceeds 4GB and ZIP's Local File Header (LFH) does not specify zip spec version 4.5

This pull request uses custom (Excel compatible) Zip64 implementation when Zip64Mode is set to Always.

[asfgit]

Can one of the admins verify this patch?

[pjfanning]

thanks - merged with https://svn.apache.org/repos/asf/poi/trunk@1861196

[bosofelipe]

@rzymek Could you give me an example because i try generate a large excel with 37000 rows and 2500 coluns and file still corrupted using apache 4.1.2

[bosofelipe]

@rzymek my current code failed

`import java.io.File;
import java.io.FileOutputStream;

import org.apache.commons.compress.archivers.zip.Zip64Mode;
import org.apache.poi.ss.usermodel.Row;
import org.apache.poi.xssf.streaming.SXSSFWorkbook;

public class TestExcel {

public static void main(String[] args) throws Exception {
	SXSSFWorkbook workbook = null;
	workbook = new SXSSFWorkbook(null, 10000, true, false);
	workbook.setZip64Mode(Zip64Mode.Always);
	int ROWS_COUNT = 37000;
	int COLS_COUNT = 2500;

	org.apache.poi.ss.usermodel.Sheet sheet = workbook.createSheet("1");
	for (int i = 1; i <= ROWS_COUNT; i++) {
		Row row = sheet.createRow(i);
		for (int j = 1; j <= COLS_COUNT; j++) {
			row.createCell(j).setCellValue("1");
			System.out.println(i + " " + j);
		}
	}

	FileOutputStream out = new FileOutputStream("~/Downloads/Excel.xlsx");
	workbook.write(out);
	out.close();
	workbook.close();
	workbook.dispose();

	File file = new File("~/Downloads/Excel.xlsx");
	System.out.println(file.length());

}

}`

[pjfanning]

Zip64Mode.AsNeeded is more correct - always may mean you use zip64 mode when you don't need it

[rzymek]

Are you getting "corrupted file" error from Excel or OpenOffice or something else? OpenOffice Calc has a limit of 1024 columns (Excel's limit is 16k columns). Other than that, the code looks ok.

Zip64Mode needs to be Always in this case to enable ZIP64 handling compatible with Excel.

[pjfanning]

@rzymek thanks for clarifying - do you know what effect setting Zip64Mode.Always has if you create a small spreadsheet - will this file cause problems for Excel?

[rzymek]

As far as I checked, Zip64Mode.Always does not cause problem with Excel even in small files. When it comes to Excel and big files (XML over 4Gb), then ZIP64 must be declared in the zip entry header before the actual zip entry contents.

[pjfanning]

Thanks @rzymek - we might want to make Zip64Mode.Always the default - needs some experimentation before we'd make that change though

[rzymek]

Exactly. I think that custom zip64 implementation should sit as an option for a few versions (it's only enabled when Zip64Mode.Always).

[bosofelipe]

@rzymek I tested with Libre Office... now i tested with MS Excel and the problem was solved, Is a limitation of Libre office, with you told us?

Tks a lot!