Ascii Codes(128-255) trigger "Binary data can't be viewed with the text editor"?

[sky5walk]

Details for the issue

I wrote 3 ansi text files as blobs to a db. Each cell in the grid displays the ascii contents. But, the text editor to the right is not consistently reading the contents.
What is the algo that triggers supposed binary content in my ascii content?
Is it size related?

Useful extra information

Each text file while viewed in Notepad++ showed as ANSI and no byte order mark(BOM).

I'm opening this issue because:

• DB4S is crashing
• DB4S has a bug
• DB4S needs a feature
• DB4S has another problem

I'm using DB4S on:

• Windows: ( version:10 Pro_ )
• Linux: ( distro: ___ )
• Mac OS: ( version: ___ )
• Other: ___

I'm using DB4S version:

• 3.10.1
• 3.10.0
• 3.9.1
• Other: ___3.10.99 nightly from 171205

I have also:

• Tried out the latest nightly version: https://github.com/sqlitebrowser/sqlitebrowser#nightly-builds
• Searched for an existing similar issue: https://github.com/sqlitebrowser/sqlitebrowser/issues?utf8=%E2%9C%93&q=is%3Aissue%20

[sky5walk]

Ok, I figured why the text & grid editor might be detecting binary content in my blob's.
Some of my files contain Ascii symbols, but they are < Ascii code 255.
Example content from blob.

1[CR][LF]
±2[CR][LF]    --<< plus/minus symbol = Ascii(177)
3°[CR][LF]    --<< degrees sign      = Ascii(176)
#END[CR][LF]

The algo must be interpreting Ascii characters > 127 as binary.
Please extend the editor up to Ascii code 255.
I want my blobs viewable to the user as text.

[justinclift]

k, this does sound like a bug that needs fixing. 😄

[mgrojo]

We are performing almost the same check in the table widget as in the cell editor, but for the first, apparently for performance reasons, we are checking only the first 512 bytes, at most, of the field.

The check is comparing the UTF-8 conversion of the data as string, with the original data. But what if the text uses another encoding? Do we always keep the text in UTF-8 or would be broken for other encodings?

@sky5walk What's your encoding? I guess it is ISO 8859-1.

[sky5walk]

As I mentioned above, my files are without a BOM and are plain Ansi text.
I will try with UTF-8 and reply.

[sky5walk]

Ok, this muddies the water a bit. All 3 cases below should be considered text by the grid and editor?

File Format      blob action
-----------      -----------
Ansi no BOM      binary
UTF8 no BOM      text
UTF8 w/BOM       binary

[mgrojo]

Have you set an encoding for the table? Does changing it help? Right-click on the table header and select "Set encoding" from the context menu. In the dialog, write "ISO 8859-1" or "CP1250" or whatever is appropriate, and then click OK.

In any case, the binary check will fail in the table widget when the first 512 bytes contain a non 7-bit-US-ASCII character.

[sky5walk]

Ok, I see the issue. There is only UTF8 or UTF16 encoding, no ANSI.
What is the reason to limit to 7 bit? Are Ascii codes 128-255 not recognized?
I type these directly into this editor and Notepad and pasted them into the DB4S editor? --> °®±
Why does it not switch to binary on paste?
I'm debating how to proceed.
Do I convert my simple ANSI doc's to UTF8 before dumping to a blob?

[mgrojo]

I think this would be a fix for the binary check in the table when the proper encoding is used. Maybe @MKleusberg can take a quick look at it for validating the solution.

@@ -871,11 +871,11 @@ void SqliteTableModel::clearCache()
 
 bool SqliteTableModel::isBinary(const QModelIndex& index) const
 {
     // We're using the same way to detect binary data here as in the EditDialog class. For performance reasons we're only looking at
     // the first couple of bytes though.
-    QByteArray data = m_data.at(index.row()).at(index.column()).left(512);
+    QByteArray data = decode(m_data.at(index.row()).at(index.column()).left(512));
     return QString(data).toUtf8() != data;
 }
 
 QByteArray SqliteTableModel::encode(const QByteArray& str) const
 {

[mgrojo]

@sky5walk The behaviour regarding encoding is probably different in Linux as in Windows. My current OS encoding is UTF-8 and DB4S is aligned with that, but don't know which is the behaviour in Windows. In my environment, DB4S expects UTF-8, which covers the full Unicode set. Any other encoding can be set in the table encoding inside DB4S, so I think there isn't any limitation to 7-bit ASCII. The unintuitive issue is knowing what encoding to set. Powerful text editors usually detect and display the used encoding when opening the text file, so one of them can help in guessing that.

[sky5walk]

Ok, it sounds like I'll store UTF8 with NO BOM. This shows as normal text in the current DB4S grid and editor. The user gets a simple browser for plain text doc's.
But, this means my blobs are not the true file contents. Meaning, I have to strip BOM's from the files prior to writing blobs to my db. I'm trying to maintain doc's and only update them when their SHA3-256 hash changes. If a doc is used and matches an existing blob's SHA3-256, then I'll refer to its ID in the table. Else, I'll write a new blob to the table and so on.
I started with plain ANSI, but that is not universal enough anymore.
Haha, now I have to contend with File SHA3's vs blob SHA3's without BOM's... or drop BOM's altogether which is another debate. Argggg.

[MKleusberg]

@mgrojo This looks like the right approach to me. We definitely need to take encoding into consideration here because currently we're assuming it is always going to be UTF-8. That's reasonable because there are no ASCII or ANSI SQLite databases but of course doesn't work for ASCII or ANSI BLOBs.
I think there are two problems though which aren't yet covered in your fix: 1) we would need to do the same in the EditDialog class which doesn't know about encoding yet, and 2) this doesn't help for UTF-8 data with BOM. It's probably a good idea to look for all possible BOMs and always assume it's text data if we find one. What do you think?

[mgrojo]

@MKleusberg The EditDialog seems to be already working (I haven't taken a look to how it's done). I've made a test database with some BLOBs in UTF-8 and some others in ISO 8859-1. Before setting the encoding, the UTF-8 work as expected, but the Latin-1 BLOBs are identified as binary, both in the table widget and the cell editor, except when the first non-US-ASCII is outside the first 512 bytes of the value, then it is identified as text by the table and as binary by the cell editor (this is expected due to the performance comprise done). When the ISO 8859-1 encoding is set for the table, the Latin-1 BLOBs are correctly identified as text in every place and the UTF-8 values are transcoded (i.e. 'á' turns 'á') as expected.

The only minor glitch is that the binary view of the decoded Latin-1 texts are showing the UTF-8 decoding and not the original binary Latin-1 encoding. This is probably messy to changge and sure it would be usually uninteresting for the user.

I think we can leave it with the decode call for the table widget that I've copied here.

The BOM part is beyond my current knowledge. I've only encountered them while working in Windows with much surprise and trouble. In Linux are apparently very unusual. In fact I don't understand why the BOM presence breaks our current check, but I suppose that ToUtf8() strips them.