Can't write xml files via "lxml" package when using pyfakefs

[buhtz]

I'm really sorry for such a broad and unspecific bug report. But I could break it down to the fact that the problem occurs only when I use pyfakefs but if I use a real fileystem for (nearly) the same test everything is fine.

Do you have any idea what could cause this side effect or how I could go on with my investigation?

Maybe pyfakefs doesn't write a real file to the fake filesystem? Can I checkt that somehow?

The last lines of the raised error

The raised errors seems to have nothing to do with pyfakefs or my own package.

  File "/usr/lib/python3/dist-packages/openpyxl/reader/excel.py", line 219, in read_worksheets
    ws = ReadOnlyWorksheet(self.wb, sheet.name, rel.target, self.shared_strings)
  File "/usr/lib/python3/dist-packages/openpyxl/worksheet/_read_only.py", line 40, in __init__
    self._get_size()
  File "/usr/lib/python3/dist-packages/openpyxl/worksheet/_read_only.py", line 46, in _get_size
    dimensions = parser.parse_dimensions()
  File "/usr/lib/python3/dist-packages/openpyxl/worksheet/_reader.py", line 164, in parse_dimensions
    for _event, element in it:
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1260, in iterator
    root = pullparser._close_and_return_root()
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1307, in _close_and_return_root
    root = self._parser.close()
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1729, in close
    self._raiseerror(v)
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1629, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: no element found: line 1, column 0

----------------------------------------------------------------------
Ran 1 test in 0.691s

FAILED (errors=1)

Description

The unittest checks if a excel file can be read. The test does this steps

1. Create a pandas.DataFrame.
2. Store it as an excel file (via pandas.DataFrame.to_excel())
3. Read the excel file into (via pandas.read_excel()).
4. Compare the initial and the returned data frame.

Of course in the real tests there happens a lot more between 2. and 3. I have a wrapper around pandas.DataFrame.to_excel().

The unittests

import unittest
import pandas
import pyfakefs.fake_filesystem_unittest as pyfakefs_ut

class Works(unittest.TestCase):

    def test_simple(self):
        """Simple excel."""
        excel_path = pathlib.Path('foobar.xlsx')
        if excel_path.exists():
            excel_path.unlink()

        df_init = pandas.DataFrame({'FOO': range(3), 'BAR': list('ABC')})
        df_init.to_excel(excel_path)

        self.assertTrue(excel_path.exists())

        df = pandas.read_excel(excel_path)

        self.assertEqual(df.shape, (3, 3))


class Problem(pyfakefs_ut.TestCase):

    def setUp(self):
        self.setUpPyfakefs(allow_root_user=False)

    def test_simple(self):
        excel_path = pathlib.Path('foobar.xlsx')
        df_init = pandas.DataFrame({'FOO': range(3), 'BAR': list('ABC')})
        df_init.to_excel(excel_path)

        self.assertTrue(excel_path.exists())

        # HERE comes the ERROR
        df = pandas.read_excel(excel_path)

        self.assertEqual(df.shape, (3, 3))

Environment

In the beginning this problem occur in older versions with Pandas (1.3.5), Numpy and openpyxl. Just for that bug report I updated everything possible (except my operating system and the python interpreter) to the current available stable release version. But the error is still there.

• Debian 11 (arm)
• Python 3.9.2 (via debian repo)
• Pandas 1.4.4 (via pip)
• Numpy 1.19.5 (via pip)
• openpyxl 3.0.10 (via pip)

Full error output

python3 -m unittest tests.test_bandas.Problem
E
======================================================================
ERROR: test_simple (tests.test_bandas.Problem)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1727, in close
    self.parser.Parse(b"", True) # end of data
xml.parsers.expat.ExpatError: no element found: line 1, column 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/ownCloud/my.work/buhtzology/tests/test_bandas.py", line 1139, in test_simple
    df = pandas.read_excel(excel_path)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/excel/_base.py", line 457, in read_excel
    io = ExcelFile(io, storage_options=storage_options, engine=engine)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/excel/_base.py", line 1419, in __init__
    self._reader = self._engines[engine](self._io, storage_options=storage_options)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/excel/_openpyxl.py", line 525, in __init__
    super().__init__(filepath_or_buffer, storage_options=storage_options)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/excel/_base.py", line 518, in __init__
    self.book = self.load_workbook(self.handles.handle)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/excel/_openpyxl.py", line 536, in load_workbook
    return load_workbook(
  File "/home/user/.local/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 317, in load_workbook
    reader.read()
  File "/home/user/.local/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 282, in read
    self.read_worksheets()
  File "/home/user/.local/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 219, in read_worksheets
    ws = ReadOnlyWorksheet(self.wb, sheet.name, rel.target, self.shared_strings)
  File "/home/user/.local/lib/python3.9/site-packages/openpyxl/worksheet/_read_only.py", line 41, in __init__
    self._get_size()
  File "/home/user/.local/lib/python3.9/site-packages/openpyxl/worksheet/_read_only.py", line 47, in _get_size
    dimensions = parser.parse_dimensions()
  File "/home/user/.local/lib/python3.9/site-packages/openpyxl/worksheet/_reader.py", line 166, in parse_dimensions
    for _event, element in it:
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1260, in iterator
    root = pullparser._close_and_return_root()
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1307, in _close_and_return_root
    root = self._parser.close()
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1729, in close
    self._raiseerror(v)
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1629, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: no element found: line 1, column 0

----------------------------------------------------------------------
Ran 1 test in 0.699s

FAILED (errors=1)

Misc

Referenced by https://codeberg.org/buhtz/buhtzology/issues/28

[mrbean-bremen]

Thanks for the report! I just had a quick local test under Windows, and the problem did not occur, so I have to make further tests under Linux. I don't have the time today (and probably not in the next few days), but I will see what I can find - right now I have no idea. From the callstack it looks like the excel file had been written in the fake fs, but the contents are incorrect. I will let you know if I find the problem, and hopefully a solution.

[mrbean-bremen]

In your related issue you write that this only happens on an ARM system. I have no possibility to test under such a system, so this may be a bit tricky...

[buhtz]

Sidenote: Different from my original report I updated Numpy to 1.23.2 which is the current stable. But nothing changed.

This becomes more wired now.

1. I'm also not able to reproduce the problem on my usual home PC running with Debian 11 (stable, amd64).
2. But I'm able to reproduce this on a RaspberyPi4 with Debian 11 (stable; ARM (arm64)).

I'm sorry. ;)

This seems not to be urgent. I assume the cause isn't pyfakefs. It must be something behind. I have to think about further tests, maybe with xmltree or openpyxl itself.

[mrbean-bremen]

Ok, thanks - in this case, I'll probably leave this alone for the time being, maybe you will find something out.

[buhtz]

Can you please have a short look on this "class".
https://github.com/lxml/lxml/blob/1e3666018329cadf8e147607824614aebf7e2099/src/lxml/serializer.pxi#L1294

My expertise end here. The file suffix is pxi and there are un-pythonic keywords like cdef. But beside of it it seems that this class tries to write files "asynchrone".

Does this give you any idea about a possible problem?

Now I have better and easier code to reproduce but I have to test it on my home machine this night first.

[mrbean-bremen]

What you are showing is lxml, a Python libray written in C, which is not compatible with pyfakefs. The keyword cdef means a function that can only be called from C code and is not visible to Python code.
I haven't seen that lxml is used in your code (xml.etree is a standard Python module and should not be a problem, and has almost the same interface as lxml), but maybe there is some OS- or hardware-specific optimization that uses it.

If lxml is used, it has to be patched away if possible, for example by replacing it with xml.etree.

[buhtz]

I haven't seen that lxml is used in your code
You are right. Iinitialy it was pandas which use openpyxl to create Excel files. And openpyxl is using lxml. ;)

[mrbean-bremen]

Hm, this still doesn't explain why it works on most systems...
The CI tests use openpyxl 3.0.10 without problems (and we don't patch it), but these tests all run under amd64 hardware (with different Linux distributions), so no surprise here.

[buhtz]

Now I can reproduce it with my home PC (amd64) and with the Pi4 (arm64).

Using lxml.etree to create and write an XML file doesn't result in a file when using a pyfakefs filesystem.

The formally used openpyxl seems to use different xml libraries if needed. See here or here. That would explain why I couldn't reproduce the problem on both machines before.

Now the interesting question is if you can reproduce it either. ;)

Code to reproduce

import unittest
import pathlib
import lxml
import lxml.etree
import pyfakefs
import pyfakefs.fake_filesystem_unittest as pyfakefs_ut


def create_file(path):
    create_xml_file(path)


def create_xml_file(path):
    with lxml.etree.xmlfile(str(path), encoding='utf-8', close=True) as xf:
        with xf.element('root'):
            xf.write(lxml.etree.Element('test'))


def read_file(path):
    read_file_xml(path)


def read_file_xml(path):
    with path.open('r') as handle:
        print(handle.read())


class Problem(pyfakefs_ut.TestCase):
    """See https://github.com/jmcgeheeiv/pyfakefs/issues/713
    """

    def setUp(self):
        self.setUpPyfakefs(allow_root_user=False)

    def test_simple(self):
        path = pathlib.Path('foobar.xlsx')
        create_file(path)

        read_file(path)

class Works(unittest.TestCase):

    def test_simple(self):
        path = pathlib.Path('foobar.xlsx')
        create_file(path)

        read_file(path)

        path.unlink()

Output of test "Works"

<root><test/></root>

Output of test "Problem"

Traceback (most recent call last):
  File "/home/user/ownCloud/my.work/pyfakefs_problem/foo.py", line 39, in test_simple
    read_file(path)
  File "/home/user/ownCloud/my.work/pyfakefs_problem/foo.py", line 20, in read_file
    read_file_xml(path)
  File "/home/user/ownCloud/my.work/pyfakefs_problem/foo.py", line 24, in read_file_xml
    with path.open('r') as handle:
  File "/home/user/.local/lib/python3.9/site-packages/pyfakefs/fake_pathlib.py", line 571, in open
    return FakeFileOpen(self.filesystem)(
  File "/home/user/.local/lib/python3.9/site-packages/pyfakefs/fake_filesystem.py", line 5721, in __call__
    return self.call(*args, **kwargs)
  File "/home/user/.local/lib/python3.9/site-packages/pyfakefs/fake_filesystem.py", line 5806, in call
    file_object = self._init_file_object(file_object,
  File "/home/user/.local/lib/python3.9/site-packages/pyfakefs/fake_filesystem.py", line 5885, in _init_file_object
    self.filesystem.raise_os_error(errno.ENOENT, file_path)
  File "/home/user/.local/lib/python3.9/site-packages/pyfakefs/fake_filesystem.py", line 1122, in raise_os_error
    raise OSError(err_no, message, filename)
FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: 'foobar.xlsx'

[mrbean-bremen]

Yes, I can reproduce this. As I wrote, lxml as a C module using C functions for file system access is not compatible with pyfakefs. This (incompatibility with Python modules using C libraries for fs access) is actually the biggest limitation of pyfakefs. From the documentation:

A module may not work with pyfakefs because of one of the following reasons:
...

• It uses C libraries to access the file system. There is no way no make such a module work with pyfakefs–if you want to use it, you have to patch the whole module. In some cases, a library implemented in Python with a similar interface already exists. An example is lxml, which can be substituted with ElementTree in most cases for testing.

[mrbean-bremen]

See also #180.

[mrbean-bremen]

Unfortunately, xml.etree.ElementTree does not have xmlfile, so substituing it won't work in your case.

[mrbean-bremen]

Ah, I missed your link to the openpyxl code that shows that it checks for availability of lxml. This is actually something we might be able to patch - just let openpyxl think that lxml is not available, so that it will use xml instead. I didn't have the time for a closer look yet (probably at the weekend), but your findings are really helpful!