fitz.Document(filename)

[JorjMcKie]

will not work if type(filename) is unicode.
Possible fix: internally convert to str(filename).

[rk700]

@JorjMcKie, I got stuck in this issue:(
It seems that I cannot convert unicode using str if the string contains non-ascii char:

>>> u = u'abcdé'
>>> type(u)
<type 'unicode'>
>>> s=str(u)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 4: ordinal not in range(128)
>>> u=u'abcde'
>>> type(u)
<type 'unicode'>
>>> s=str(u)
>>> s
'abcde'

I read the python doc and it says the encoding should be provided, otherwise ASCII is used by default and a character with an ASCII value > 127 in the input data will result in UnicodeEncodeError.

Any ideas?

[JorjMcKie]

Dear Ruikai,

First, I don’t think it’s a big issue.

Acceptable filenames (at least in windows) are dependant on the locale of the installation anyway. So, on an US computer something like jubilé.pdf would not be accepted, on my Windows it would.

So, specifying acceptable filenames entirely is the end user’s responsibility.

At the moment, the interface requests, that the f in fitz.Document(f) must formally be of type str – no matter in what encoding.

If that requirement could be changed into unicode, all would be fine.

I am far from being a C expert, so I don’t know how do that in C.

The SWIG documentation in the Python chapter on the other hand says, that all strings coming from SWIG are utf-8, and it is the user’s responsibility to convert them when needed.

For example, both of the following stament sequences work on my computer and create the same file:

f = u“jubilé.txt“

fle = open(f,“w“)

fle.close()

and

f = u“jubilé.txt“.encode(“utf-8“)

fle = open(f,“w“)

fle.clode()

So, another way could be to just put in the (future) documentation, that “filename must be a utf-8 string like all other strings dealing with MuPDF“: e.g. the text output of the MuPDF tool “mutool.exe“ is strings in utf-8 encoding – and we do not need tob e any better :).

I guess that’s all I can say.

Best regards

Jorj

Von: Ruikai Liu [mailto:notifications@github.com]
Gesendet: Dienstag, 26. Mai 2015 21:54
An: rk700/python-fitz
Cc: Jorj X. McKie
Betreff: Re: [python-fitz] fitz.Document(filename) (#8)

@JorjMcKie https://github.com/JorjMcKie , I got stuck in this issue:(
It seems that I cannot convert unicode using str if the string contains non-ascii char:

u = u'abcdé'
type(u)
<type 'unicode'>
s=str(u)
Traceback (most recent call last):
File "", line 1, in
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 4: ordinal not in range(128)
u=u'abcde'
type(u)
<type 'unicode'>
s=str(u)
s
'abcde'

I read the python doc and it says the encoding should be provided, otherwise ASCII is used by default and a character with an ASCII value > 127 in the input data will result in UnicodeEncodeError.

Any ideas?

—
Reply to this email directly or view it on GitHub #8 (comment) . https://github.com/notifications/beacon/AH6BoofsXmL9JodVKflhAkDy9dZ6zA9lks5oNSI5gaJpZM4EqzXW.gif

[rk700]

@JorjMcKie, hmm, since it's ok to pass non-ascii string to python open, maybe we can internally create a python file object and convert that to a C FILE *. Some fitz IO functions also support passing a FILE * as argument. I'll try and see if it works.

[rk700]

@JorjMcKie, fz_new_document doesn't support FILE pointer, so I add a conversion from unicode to str. https://github.com/rk700/python-fitz/blob/master/fitz/fitz.i#L37

[JorjMcKie]

ok. I will test that as soon as possible then.

[JorjMcKie]

Solution works with all of my test cases - closing the issue.

[ashishelskiy]

Hello! Can you tell me how to solve this problem?
mupdf: bad data in ahxd: '�'
mupdf: cmsOpenProfileFromMem failed

[ashishelskiy]

This is the code which gives an error.

import fitz

doc = fitz.open('./some.pdf')

with open('./some.txt','w') as f:
for current_page in range(len(doc)):
page = doc.loadPage(current_page)
page_text = page.getText("text")
f.write(page_text)

[JorjMcKie]

looks like file "some.pdf" has an internal problem:

/**
	ahxd filter performs ASCII Hex decoding of data read
	from the chained filter.
*/

It seems to contain invalid ASCII encoded data.
You could try to first clean and decode / decompress the file before trying to extract text: e.g. mutool clean -d some.pdf.

[ashishelskiy]

Hello, Jorj! Thanks for the quick response! After cleaning, another error "mupdf: cmsOpenProfileFromMem failed".

[JorjMcKie]

There is some issue with the ICC color management. Apparently you can simply ignore it - at least for the purpose of text extraction.
Maybe some colors do not look right when you make pixmaps, or so.
Look at the following snippet - after each statement, I am showing and resetting the full message stack of MuPDF (errors and warnings):

>>> import fitz
>>> doc=fitz.open("tjm-1.PDF")
>>> print(fitz.TOOLS.mupdf_warnings())

>>> page=doc[0]
>>> print(fitz.TOOLS.mupdf_warnings())

>>> text=page.get_text()
mupdf: cmsOpenProfileFromMem failed
>>> print(fitz.TOOLS.mupdf_warnings())
bad data in ahxd: 'î'
read error; treating as end of file
lcms: Couldn't allocate 0 bytes for profile.
cmsOpenProfileFromMem failed
ignoring broken ICC profile
>>> 

Of the 5 lines contained in the message stack, the second-to-last is flagged as error message, the others are warnings (which are suppressed from display).
You can suppress error messages too if they molest you (has some risk obviously): fitz.TOOLS.mupdf_display_errors(False).
The error message here is only show once anyway.

[ashishelskiy]

Hi, Jorj! Sorry for the long answer! I saw your answer immediately, but I did not have time to answer you. Yes, I already understood earlier that these are only error messages and do not affect anything. You have first class tech support, thank you!