-
-
Notifications
You must be signed in to change notification settings - Fork 326
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't handle files >=4GB #1713
Comments
This will be only the first triggered problem (out of many other problems I expect for files larger than 2 GB). Support from Scintilla is also in beta state (see also: https://www.scintilla.org/ScintillaDoc.html#MultipleViews : SC_DOCUMENTOPTION_TEXT_LARGE). Beside the expected "sluggish" behavior of the |
I think at least a warning and failing or something similar about this would be nice, just in case someone tries to open large files like this. Additionally, you might want to check smaller sizes too, like 4GB-16 (which ends up at an allocation size of 0 bytes) to make sure they don't crash the program. Appropriately sized files also bypass the file size warning currently. (also just a sidenote: my file was in fact a sensible text document, namely a log file) |
It's not slow when use idle styling, see zufuliu/notepad4#125. |
HI @RaiKoHoff, there are bugs in WideCharToMultiByteEx() and MultiByteToWideCharEx(): they don't handle UTF-16 surrogate pairs (CharNextW, CharPrevW) or DBCS multibyte characters (IsDBCSLeadByteEx, CharNextExA, CharPrevExA). |
Hi @zufuliu , thanks for the hint, it was just an idea workaround the 2GB (MAX_INT) limitation, I didn't test or used it yet ... 🤔 |
Hello @namazso , for very large files, I recommend for example: PilotEdit 13.3.0 PilotEdit is a handy and reliable file (text- and hex-) editor designed to help users to execute scripts, extract strings and edit large files. Features:
|
Hello @RaiKoHoff , The file: The file:
|
@hpwamr how about Notepad2, you can test the AVX2 build. |
Hello @zufuliu , indeed your AVX2 build is very fast to open a big file. |
use SciCall_SetIdleStyling(SC_IDLESTYLING_ALL) or with other none SC_IDLESTYLING_NONE value. and don't call SciCall_ColouriseAll() in Style_SetLexer(), call it somewhere when colourising whole document is needed, i.e. when toggling all folds. SciTE use SC_DOCUMENTOPTION_STYLES_NONE for bigger file (default 10MB). |
@zufuliu : Sorry, we didn't find the time to look at your solution yet - I think too: that is the right way. |
@RaiKoHoff basic changes
About "file mapping": I think it's more complicated, I don't work out how to avoid loading entire file into memory but still keep correct syntax color and folding (all file in my Notepad2 has code folding). I only have experience (for viewing huge SQL dump) with Vim which can load file larger than 4GB with color (seems Vim loads file chunks on demand, but I not checked it's source code). |
Just tested file mapping (based on example from https://docs.microsoft.com/en-us/windows/win32/memory/creating-a-view-within-a-file), mapping entire file is indeed faster than reading entire file, the other benefit of file mapping is that the API supports file larger then 4GiB. |
@zufuliu : Exactly what I expected: Creating only a Mapped View on huge files should be faster 😃. On the other hand, I have to dig deeper in that issue, maybe Notepad3 may not support a huge encrypted file (if it uses a stream cipher ...). |
@RaiKoHoff searching is OK, including mark all occurrences. Editing (a rarely operation for huge files) is very slow, with huge memory/CPU usage (resizing and moving). Saving is not slow (using WriteFile), using file mapping (and range pointer) to save file is not tested, I think it should faster than WriteFile. Any way, loading entire file into memory is not the proper way to view huge files, but it's the only way supported by current Scintilla. |
Hello @hpwamr , Feel free to test the RC2 version "Notepad3Portable_5.20.225.3_RC2.paf.exe.7z" or higher. Note: "Notepad3Portable RC2" can be used in "2 flavors", see with/without ext.: ".7z" or from site_2. Your comments and suggestions are welcome... 😃 |
Hello @namazso , Today, I've tested
The file is opened after +/- 8 sec on a fast i7 system. Note: The maximum size is changed from 2 GB to 4 GB. 👍 As far as I'm concerned, I think you (requester) can close this issue... |
@hpwamr well, the original issue was about an 5.76 GB file, so I think it could stay. however I appreciate the increased limit. |
We can leave this issue open, until the Scintilla Component can handle files bigger than 4GB ... |
Hello @zufuliu , Download test files: "Test_big_files_till_4GB.rar" The test file:
|
Hi @hpwamr, thank you 👍. What's about the time for AVX2 builds (v4.20.08r3252, v4.20.09r3288)? For Notepad3, there will have improvements after Scintilla 5 (maybe in 5.1 or 5.2). |
@zufuliu : Thank you for enhancement suggestions - I will check your latest code for EOL detection soon... - best regards 👍 |
Hello @zufuliu , |
@hpwamr Thanks. |
I tried opening a file (5.76 GB) with Notepad3 5.19.815.2595 (x64), however it stopped displaying the file at around 1.76 GB into the file. Search or anything else also doesn't work past that point.
the problem appears to be here:
Notepad3/src/Edit.c
Line 983 in 68f0e44
because the higher DWORD of the file size is ignored
The text was updated successfully, but these errors were encountered: