pdfminer can't parse characters outside the ASCII encoding

[marianorodriguez]

Attached is a document in spanish that shows that pdfminer cant process latin characters like:
á, é, í, ó, ú, ñ, etc...

caixa-one-page-spanish.pdf

[peter-vandenabeele-axa]

Sorry, I fail to understand the title, relative to the problem statement.

I understand this as "pdfminer can't parse characters outside the ASCII encoding"
(because the "accent" characters from French and Spanish etc. that you refer to are part of utf-8).

Probably I am just misunderstanding ?

[royjohal]

Attached is a document in spanish that shows that pdfminer cant process latin characters like:
á, é, í, ó, ú, ñ, etc...

It's actually more complicated than that.
Also: this behavior can also manifest on standard latin characters: it all depends on how the PDF document was encoded - a PDF can lack a complete textual mapping by omission since it is not an inherent purpose of the PDF format to provide the textual equivalent.

Lines producing the '?':

Parsr/server/src/input/pdfminer/pdfminer.ts

Lines 196 to 205 in 6586953

	/**
	* Fetches the character a particular pdfminer's textual output represents
	* TODO: This placeholder will accomodate the solution at https://github.com/aarohijohal/pdfminer.six/issues/1 ...
	* TODO: ... For now, it returns a '?' when a (cid:) is encountered
	* @param character the character value outputted by pdfminer
	* @param font the font associated with the character -- TODO to be taken into consideration here
	*/
	function getValidCharacter(character: string): string {
	return RegExp(/\(cid:/gm).test(character) ? '?' : character;
	}

Related Issues:

1. pdfminer does not always produce clean textual output #75
2. Find out equivalent character replacements for (cid:nn) characters royjohal/pdfminer.six#1
3. Japanese characters shown as (cid:3821) etc. pdfminer/pdfminer.six#130

[royjohal]

An interesting related patent: https://patents.google.com/patent/US20060288281

[royjohal]

This is a general problem related to the extractors used, instead of being particularly Parsr's problem.