Searching files without UTF-8 BOM?

[Liturgist]

Select Topic Area

Question

Body

Documentation says that only UTF-8 encoded files are searched.

Does that mean that a UTF-8 BOM (Byte Order Mark) is required at the beginning of the file? It does not appear so.

Is a file omitted from search if a non-UTF-8 character sequence is found?

[look]

Is a file omitted from search if a non-UTF-8 character sequence is found?

That's correct. We convert the raw bytes to UTF-8 and if the conversion is not successful, the blob is excluded.

[Liturgist]

This seems to suggest that files without a Unicode BOM are searched. Is that correct?

How is it determined what the incoming encoding is? Converting from ??? to UTF-8.

[look]

The files are validated with Go's utf8.Valid(). This checks the byte sequence to ensure it is valid UTF-8. Non UTF-8 encodings are not transcoded. I validated that this function does work with a byte order mark, though these are generally not recommended for UTF-8 encoded files.