Update SQLPlatform.php #18
Conversation
prevent indexing of drawio files as well as files where content is not pure utf8 compatible
| $content = $pdf->getText(); | ||
| } elseif ( | ||
| str_starts_with($content, "<mxfile ") // draw.io file | ||
| || !mb_detect_encoding($content, "UTF-8", true) // binary file |
There was a problem hiding this comment.
Indexing non-UTF8 is a feature, not a bug. 😉 See the call to mb_convert_encoding() below. (Many of - at least my - PDFs have non-UTF8 text in it)
There was a problem hiding this comment.
Come to think of it, that call could be modified to explicitly call mb_detect_encoding() with $strict = true first. If the system setting for what encodings to detect then only contains UTF-8, that would return false and you could abort at that point.
There was a problem hiding this comment.
The idea was the following:
- identify pdf files
- identify drwa.io files ("cheap" test, only check the first few bytes)
- identify files which are no "text" files ("expensive", since it might be necessary to scan all data)
Just reject that merge request and implement it the way fitting your design.
I'm not good in php, neither do I know the complete return value signatures, hence using a shortcut and "wiping" the content to be stored.
There was a problem hiding this comment.
Okay, I've implemented at least the binary part in 0e794a1
2 participants
prevent indexing of drawio files as well as files where content is not pure utf8 compatible
potential fix for #15