QGIS crashes/freezes when adding a large CSV file as delimited text layer #36392
Comments
fet760
added
the
Bug
@fet760 just ZIP your CSV and attach it here (github takes ZIPs). |
|
@fet760 also try with a new QGIS profile. |
|
By new QGIS profile do you mean new project? If so, I have done that
multiple times starting from scratch, everything goes well until that point
where it crashes.
…On Tue, May 12, 2020 at 7:12 PM Giovanni Manghi ***@***.***> wrote:
@fet760 <https://github.com/fet760> also try with a new QGIS profile.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#36392 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/APR5AMKRKDOFUAGRBP2C6DDRRHQWZANCNFSM4M7IY27A>
.
|
|
@fet760 settings > user profiles > new profile |
|
[image: image.png]Tried using profile Fet760 and TEST123, both profiles had
the same issue.
…On Tue, May 12, 2020 at 7:20 PM Giovanni Manghi ***@***.***> wrote:
@fet760 <https://github.com/fet760> settings > user profiles > new profile
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#36392 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/APR5AMLAFUSJWZL5VMQTWRDRRHRUJANCNFSM4M7IY27A>
.
|
|
@fet760 please attach the original CSV file (as zip). |
|
in my case it causes a freeze rather than a crash, but... the CSV is mostly made of empty cells (of about 1 million lines only ~300 contain data). If you isolate the lines with data it opens with no issues. Now it must e seen if the problem are the empty lines or if is a more general issue with CSVs with so many (empty or not) lines. |
|
could you guide me on how to isolate the lines with data. |
open the CSV in a robust text editor (not notepad, if you are on Windows use Notepad++) select the lines with data, copy and paste in a new text file with .csv extension. |
|
I selected all the blank cells and deleted them. The file went from 11MB to 27KB. It works now. Thanks for your time and help sir hope you have a great rest of the day. |
|
Reopening as the issue remains. |
gioman
changed the title
Fixes qgis#36392 Fixes qgis#21976 Fixes qgis#17190 We are obliged to do 'at hand' parsing due to QT not handling CR-only end of lines. As we are at it, also limit each line to 1 MB to avoid potential denial of service (which was what close to what happened here before the CR-only parsing fix) Add tests for parsing CR-only end of lines, and exercising the at-hand buffering logic
Set to 10000 by default. Can be overriden through settings. This is only for the preview in the source select dialog. (Attribute table performs badly for much less columns) Related to qgis#36392
Fixes #36392 Fixes #21976 Fixes #17190 We are obliged to do 'at hand' parsing due to QT not handling CR-only end of lines. As we are at it, also limit each line to 1 MB to avoid potential denial of service (which was what close to what happened here before the CR-only parsing fix) Add tests for parsing CR-only end of lines, and exercising the at-hand buffering logic
Set to 10000 by default. Can be overriden through settings. This is only for the preview in the source select dialog. (Attribute table performs badly for much less columns) Related to #36392
Fixes qgis#36392 Fixes qgis#21976 Fixes qgis#17190 We are obliged to do 'at hand' parsing due to QT not handling CR-only end of lines. As we are at it, also limit each line to 1 MB to avoid potential denial of service (which was what close to what happened here before the CR-only parsing fix) Add tests for parsing CR-only end of lines, and exercising the at-hand buffering logic (cherry picked from commit 644a564)
Set to 10000 by default. Can be overriden through settings. This is only for the preview in the source select dialog. (Attribute table performs badly for much less columns) Related to qgis#36392 (cherry picked from commit fcf6203)
Fixes #36392 Fixes #21976 Fixes #17190 We are obliged to do 'at hand' parsing due to QT not handling CR-only end of lines. As we are at it, also limit each line to 1 MB to avoid potential denial of service (which was what close to what happened here before the CR-only parsing fix) Add tests for parsing CR-only end of lines, and exercising the at-hand buffering logic (cherry picked from commit 644a564)
|
Same thing happens in 3.18, opening the attribute table for large point layer (few millions points) freezes and never finishes. |
@pocwiard try QGIS master, I think there has been a fix recently (but I may be wrong). |
|
@gioman thanks. I tried to locate how to download the master but I failed miserably. Is there an easy way to download the master? |
|
|
@gioman thank you very much! |
Operating system: Windows 10 Pro (Version 1909)
QGIS Version: 3.12
Attaching the xlsx version of the csv file as github doesnt support csv files.
I was not having this issue until today. Started using QGIS version 3.10 (more stable) last week and this morning was working fine until noon where it started crashing when uploading this delimited text layer. I restarted my pc, nothing, unistalled version 3.10 and installed version 3.12 but the problem persisted. Thanks for any type of help in advance.
ATT.xlsx
The text was updated successfully, but these errors were encountered: