invalid byte sequence in UTF-8 (ArgumentError)

[y0608]

Trying to extract text from this file, I get the error: invalid byte sequence in UTF-8 (ArgumentError)

It fails to extract the text from the 3rd page.

ruby -Ilib bin/pdf_text ~/Downloads/garage_remote.pdf

returns:

...text...

/home/mint/Programming/open_source/pdf-reader/lib/pdf/reader/page_layout.rb:75:in `strip': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/mint/Programming/open_source/pdf-reader/lib/pdf/reader/page_layout.rb:75:in `block in interesting_rows'
        from /home/mint/Programming/open_source/pdf-reader/lib/pdf/reader/page_layout.rb:75:in `map'
        from /home/mint/Programming/open_source/pdf-reader/lib/pdf/reader/page_layout.rb:75:in `interesting_rows'
        from /home/mint/Programming/open_source/pdf-reader/lib/pdf/reader/page_layout.rb:52:in `to_s'
        from /home/mint/Programming/open_source/pdf-reader/lib/pdf/reader/page.rb:140:in `text'
        from bin/pdf_text:12:in `block in <main>'
        from bin/pdf_text:11:in `each'
        from bin/pdf_text:11:in `<main>'

Is this output expected? And if so, how would you recommend handling it?

[yob]

This output is a bug that we should handle better.

I tested extracting the text of page 3 using pdftotext, and the relevant section includes:

<U+D8B1> Actuator AZIMUT

cable with plug 2 x 0.75 mm2

<U+D8B2> Photocells

cable 4 x 0,5 mm2 (RX)
cable 2 x 0,5 mm2 (TX)

<U+D8B3> Key switch

U+D8B1 is not a valid unicode codepoint to exist on its own, so this PDF has some invalid encoding in it. pdf-reader should handle that more gracefully though, probably be replacing the invalid codepoint with a ▯ or �

[yob]

I don't have a fix committed yet, but the root issue is this difference table in the PDF:

<</Differences [2 /L55473 /L55474 /L55475 /L55476 /L55477 /L55478 ] /Type /Encoding /BaseEncoding /MacRomanEncoding >>

PDF Names like /L55473 are being mapped to Unicode U+55473 (AKA U+D8B1), which is invalid on its own.

This "fixes" the exception in pdf-reader by detecting invalid utf8 in difference tables and replacing the character with ▯:

diff --git a/lib/pdf/reader/encoding.rb b/lib/pdf/reader/encoding.rb
index ceb3df1..46722cb 100644
--- a/lib/pdf/reader/encoding.rb
+++ b/lib/pdf/reader/encoding.rb
@@ -167,6 +167,9 @@ class PDF::Reader
         @mapping[glyph_code.to_i] || glyph_code.to_i
       ].pack("U*")
       ret.force_encoding("UTF-8")
+       unless ret.valid_encoding?
+         ret = [UNKNOWN_CHAR].pack("U")
+       end
       ret
     end

I'll work on building a sample file with issue and a proper fix with a test