-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[QUESTION] Reading huge excel file consumes too much memory. Caused by simplexml_load_string #2793
Comments
Depending on your server memory size, you can increase the memory limit of the process (ini set)
As you said, it happens inside PhpSpreadsheet, which means there's nothing we can do in this package unfortunately. |
I'm running my import command on a server with small amount of memory to save costs. It's possible to temporary switch to a server with large amount of memory, it's just not very convenient to do so every time I run import comand. Limiting memory with php's memory_limit option does not work for memory allocations done by SimpleXML. If php is running inside a container, memory limit enforced by container runtime works of course, and kills the process to enforce limit. I just wanted to confirm that there isn't a way to solve the problem without rewriting my program to use another package. |
Hello @nbyloff, Did you find something else to achieve this? It seems that fast-excel also has issues with large imports. Thanks. EDIT: it seems that I have the same issue using CSV. Unfortunately, XLS is limited to 65k~ rows. |
@mtx-z I came up with a solution that didn't involve spreadsheets that large; they are much shorter now. However, the only way I got it to work (before I abandoned it) was to make it a two step process. First I would write all my data to a DB table that tracked an excel row id & an excel column id. Step two was to use a cursor to query the DB (sorting by row id & column id) and write everything in sequence (A1, B1, C2, A2, B2, C2, etc). I also had been doing it with this library as it was the fastest I found. But that is now archived. If you are writing the data in sequentially in Excel (and not jumping around to cells writing data), it might work with a different library. I essentially learned the memory issues came from jumping around to different cells too much in these massive spreadsheets. It had to do it in order so there was nothing to track in memory. |
@nbyloff thank you for your answer. I ended up doing the same as you: I imported my Excel file into the database, and imported it from a custom job using Fast-Excel is a Laravel wrapper for Spout that you linked, so my guess is the large file import issue I linked occurs with both. Thank you again for those details and your tests. |
Prerequisites
Versions
Description
Reading huge excel file (over 10MB) causes php process to be killed. The sheet causing the issue is over 75MB after uncompressing the file.
I tracked down the place causing the error, and found that simplexml_load_string consumes around 2GB of memory.
Is there a way to reduce memory usage?
https://github.com/PHPOffice/PhpSpreadsheet/blob/9683e5be189080a9c75298e00efb72be3f5c866a/src/PhpSpreadsheet/Reader/Xlsx.php#L621
This only occurs with XLSX files because other types of files does not use SimpleXML for reading.
Chunked reading does not help because the problems occurs before laravel excel starts reading rows.
The problem is happening inside PhpSpreadsheet library, and SimpleXML is not suited for reading huge file as it tries to load entire tree into memory.
PhpSpreadsheet does not probably support other types of XML parses which uses less memory, so there is nothing I can do as long as I use Laravel Excel?
The text was updated successfully, but these errors were encountered: