.vtu file parsing errors after libexpat1 upgrade (Ubuntu 22.04, 23.10, 24.04)

[ptheywood]

Due to security fixes for libexpat1/libexpat-dev(between 2024-03-12 and 2024-03-18), tests which require reading .vtu files which contain binary data now fail for up to date ubuntu 22.04 users (and potentially other distributions too).

This has been fixed upstream by VTK (db8f9efc) but this has not yet been backported to Ubuntu.

E.g. for the Continuous test suite

$ ctest -L Continuous
...
The following tests FAILED:
         78 - TestQuadraticMesh (Failed)
         93 - TestVtkMeshReader (Failed)
        113 - TestXmlMeshWriters (Failed)
        175 - TestHdf5Converters (Subprocess aborted)
        262 - TestNodeBasedCellPopulation (Failed)
        290 - TestOnLatticeSimulationWithPottsBasedCellPopulation (Failed)
        367 - TestMajorAirwaysCentreLinesCleaner (Failed)
        382 - TestBidomainProblem (Failed)
        420 - TestMonodomainProblem (Failed)
        432 - TestPostProcessingWriter (Subprocess aborted)

A short-term workaround for ubuntu 22.04 users is to downgrade libexpat1 and libexpat1-dev packages, however, this is not an ideal soluation as it removes vulnerability patches:

sudo apt install libexpat1=2.4.7-1 libexpat1-dev=2.4.7-1
sudo apt-mark hold libexpat1 libexpat1-dev

23.10 (Mantic) users may be able to downgrade to 2.5.0-2 (untested).
24.04 users do not appear to have an older version to downgrade to.

As of #255, this workaround has been applied to CI for 22.04 based workflows and containers.

An additional CI workflow has been set up to run every night without this patch, to monitor when this is resolved upstream (likely via a paraview/vtk update?). Once this new workflow passes, the github bot will leave a comment on this issue stating as much, before disabling the workflow.

When this occurs we can:

• Revert the changes made in CI libexpat1 downgrade #255, using up to date libexpat1.
• Revert the changes made to Chaste/chaste-docker in Chaste/chaste-docker@947d10e and Chaste/chaste-docker@d4d011a
• Unhold libexpat1 and libexpat1-dev on np3.
  • apt-mark unhold libexpat1 libexpat1-dev
• Close this issue.

Relevant issues:

• Ubuntu VTK7 Bug report
• VTK Gitlab Issue
• Paraview Discourse thread

---

Ubuntu 22.04 CI failures starrted to occur between 2024-03-12 and 2024-03-18.

This is due to a libexpat1 update to 2.4.7-1ubuntu0.3, and has been reported upstream.

https://bugs.launchpad.net/ubuntu/+source/expat/+bug/2058415

Locally, downgrading to 2.4.7-1 has resolved these issues for now, via:

sudo apt install libexpat1=2.4.7-1 libexpat1-dev=2.4.7-1

But this will also need holding/pinning to avoid being re-upgraded (e.g. apt-mark hold libexpat1 libexpat1-dev).
This could be applied on CI, as a short term workaround, but the issue will still be present for other Ubuntu 22.04 users until this is fixed upstream.

---

For example TestQuardraticMesh fails as follows:

ctest -R TestQuadraticMesh --output-on-failure
Test project /home/ptheywood/code/chaste/Chaste/build-clang-12
    Start 78: TestQuadraticMesh
1/1 Test #78: TestQuadraticMesh ................***Failed    0.74 sec
Running 23 tests

 ***** TestQuadraticMesh.hpp *****


Entering TestQuadraticMesh1d
Passed
Entering TestQuadraticMesh1dAutomaticallyGenerated
Passed
Entering TestQuadraticMesh2d
Passed
Entering TestQuadraticMesh3d
Passed
Entering TestAutomaticallyGenerated2dMesh1
Passed
Entering TestAutomaticallyGenerated2dMesh2
Passed
Entering TestAutomaticallyGenerated3dMeshSimple
Passed
Entering TestAutomaticallyGenerated3dMesh
Passed
Entering TestWritingReadingBoundaryElementsWithContainingElementInfo
Passed
Entering TestArchiving
Passed
Entering TestConstructRegularSlabMesh_Directly_1d
Passed
Entering TestConstructRegularSlabMesh_Directly_2d
The event associated with the counter for 'face write' had already begun when BeginEvent was called.
Passed
Entering TestConstructRegularSlabMesh_Directly_3d
Passed
Entering TestConstructionConversionVersusConstruction2dNoStagger
Passed
Entering TestConstructionConversionVersusConstruction2dWithStagger
Passed
Entering TestConstructionConversionVersusConstruction3d
Passed
Entering TestLinearToQuadraticMeshConversion2d
Passed
Entering TestLinearToQuadraticMeshConversion2dNonconvex
Passed
Entering TestLinearToQuadraticMeshConversion3d
Passed
Entering TestLinearToQuadraticMeshConversion3dNonconvex
Passed
Entering TestQuadraticMesh2dReordered
Passed
Entering TestBuildQuadraticMeshFromVtkMeshReader
ERROR: In ./IO/XMLParser/vtkXMLParser.cxx, line 389
vtkXMLDataParser (0x2119460): Error parsing XML in stream at line 25, column 15, byte index 1315: not well-formed (invalid token)

ERROR: In ./IO/XML/vtkXMLReader.cxx, line 467
vtkXMLUnstructuredGridReader (0x21105c0): Error parsing input file.  ReadXMLInformation aborting.

ERROR: In ./Common/ExecutionModel/vtkExecutive.cxx, line 781
vtkCompositeDataPipeline (0x21283e0): Algorithm vtkXMLUnstructuredGridReader(0x21105c0) returned failure for request: vtkInformation (0x22670c0)
  Debug: Off
  Modified Time: 83
  Reference Count: 1
  Registered Events: (none)
  Request: REQUEST_INFORMATION
  ALGORITHM_AFTER_FORWARD: 1
  FORWARD_DIRECTION: 0



Error:  The point set is trivial.

/home/ptheywood/code/chaste/Chaste/mesh/test/TestQuadraticMesh.hpp:884: Error: Test failed: Exception thrown from test
Failed
Entering TestElementsContainedByNodes3d
Passed

Failed 1 of 23 tests
Success rate: 95%


0% tests passed, 1 tests failed out of 1

Label Time Summary:
Continuous_mesh    =   0.74 sec*proc (1 test)

Total Test time (real) =   0.77 sec

The following tests FAILED:
         78 - TestQuadraticMesh (Failed)
Errors while running CTest

I've managed to narrow this down to the upgrade of either libexpat1-dev or libexpat1, as I had an updated and an not-updated Ubuntu 22.04 machine available (with ~100 packge updates difference).

from apt list --upgradable:

libexpat1-dev/jammy-updates,jammy-security 2.4.7-1ubuntu0.3 amd64 [upgradable from: 2.4.7-1ubuntu0.2]
libexpat1/jammy-updates,jammy-security 2.4.7-1ubuntu0.3 amd64 [upgradable from: 2.4.7-1ubuntu0.2]

After updating these packages via sudo apt install libexpat1-dev libexpat1 and performing a clean build the following test failures then occur:

$ ctest -L continuous
...
The following tests FAILED:
         78 - TestQuadraticMesh (Failed)
         93 - TestVtkMeshReader (Failed)
        113 - TestXmlMeshWriters (Failed)
        175 - TestHdf5Converters (Subprocess aborted)
        262 - TestNodeBasedCellPopulation (Failed)
        290 - TestOnLatticeSimulationWithPottsBasedCellPopulation (Failed)
        367 - TestMajorAirwaysCentreLinesCleaner (Failed)
        382 - TestBidomainProblem (Failed)
        420 - TestMonodomainProblem (Failed)
        432 - TestPostProcessingWriter (Subprocess aborted)

[ptheywood]

The following could be added to CI workflows to perform this downgrade just from this exact version, for workflows where matrix.os is defined.

The alternative when this is using a container is untested, but the upgrade will only occur if the offending package is installed. Non debian-based workflows would need another condition adding in place.

    - name: Downgrade libexpat
      if: ${{ (runner.os == 'Linux' && matrix.os == 'ubuntu-22.04') || job.container != null }}
      run: |
        if dpkg -l | grep libexpat1-dev | grep 2.4.7-1ubuntu0.3; then
          sudo apt-get install -y --allow-downgrades libexpat1=2.4.7-1 libexpat1-dev=2.4.7-1
        fi

[ptheywood]

A few more relevant links to keep an eye on for fixes. It seems to be that paraview is going to need to be patched for .vtu parsing to work again.

• https://discourse.paraview.org/t/i-cannot-read-a-vtp-file-i-could-open-yesterday-can-someone-try-to-open-it/13938/22
• https://gitlab.kitware.com/vtk/vtk/-/issues/19258
• Debian and Ubuntu packages failing with version 2.6.2 libexpat/libexpat#857

As this is effected by distribution specific patches/backports, it'll be hard to guard against / detect this programatically.

Potenitally could detect it at cmake configuration time via try_run to build a tiny program which opens a vtu file containing binary data, check its return code and then warn / error if this fails. Although suggeting users undo a security fix doesn't seem ideal.

[ptheywood]

Detection via try_run is possible, but VTK does not make it trivial to programatically detect XML reading/parsing errors, instead just printing them to stdout and continuing.

I've pushed a crude first stab at this to libexpat1-detection, which currently will cause cmake configuration to error if a vtu file cannot be parsed, with an unfinished, overly long error message.

$ cmake .. 
...

CMake Error at CMakeLists.txt:403 (message):
  .vtu parsing runtime error.  probably bad libexpat1? @todo - helpful
  message ERROR: In ./IO/XML/vtkXMLReader.cxx, line 467

  vtkXMLUnstructuredGridReader (0x55f7d7f840d0): Error parsing input file.
  ReadXMLInformation aborting.



  ERROR: In ./Common/ExecutionModel/vtkExecutive.cxx, line 781

  vtkCompositeDataPipeline (0x55f7d7f84e50): Algorithm
  vtkXMLUnstructuredGridReader(0x55f7d7f840d0) returned failure for request:
  vtkInformation (0x55f7d7f8b5f0)

    Debug: Off
    Modified Time: 85
    Reference Count: 1
    Registered Events: (none)
    Request: REQUEST_INFORMATION
    ALGORITHM_AFTER_FORWARD: 1
    FORWARD_DIRECTION: 0

  







-- Configuring incomplete, errors occurred!

This took a lot more effort than I'd hoped, but something to conisder / discuss if this wants taking further. Ideally VTK/paraview will be patched where required and this won't actually be neccessary long term.

Commit 7003517 includes the crude (incomplete but functional) demonstration.

[ptheywood]

Todo from in-meeting discussion.

• Patch CI workflows where needed
• Add temporary libexpat1 checking ci workflow that will fail while libexpat1 / is broken. This will fail.
  • Could make this a scheduled job that does something when it's fine?. So regular CI will be all ticks, but there will be a non-pr associated failure?
• Open issue to remove libexpat1 ci fix / tempoarary workflow when upstream fixes things

[bdevans]

Hi,
I've tried adding the downgrade to the chaste-docker base image for both arm64 and amd64 on jammy but the build fails:

 > [linux/amd64 base  7/17] RUN apt-get install -y --allow-downgrades libexpat1=2.4.7-1 libexpat1-dev=2.4.7-1 &&     apt-mark hold libexpat1 libexpat1-dev &&     rm -rf /var/lib/apt/lists/*:
0.497 Package libexpat1 is not available, but is referred to by another package.
0.497 This may mean that the package is missing, has been obsoleted, or
0.497 is only available from another source
0.497 
0.497 Package libexpat1-dev is not available, but is referred to by another package.
0.497 This may mean that the package is missing, has been obsoleted, or
0.497 is only available from another source
0.497 
0.497 E: Version '2.4.7-1' for 'libexpat1' was not found
0.497 E: Version '2.4.7-1' for 'libexpat1-dev' was not found

(The full job output is here.)
Do you have any idea why it can't find the older files?

[ptheywood]

Hi, I've tried adding the downgrade to the chaste-docker base image for both arm64 and amd64 on jammy but the build fails:

 > [linux/amd64 base  7/17] RUN apt-get install -y --allow-downgrades libexpat1=2.4.7-1 libexpat1-dev=2.4.7-1 &&     apt-mark hold libexpat1 libexpat1-dev &&     rm -rf /var/lib/apt/lists/*:
0.497 Package libexpat1 is not available, but is referred to by another package.
0.497 This may mean that the package is missing, has been obsoleted, or
0.497 is only available from another source
0.497 
0.497 Package libexpat1-dev is not available, but is referred to by another package.
0.497 This may mean that the package is missing, has been obsoleted, or
0.497 is only available from another source
0.497 
0.497 E: Version '2.4.7-1' for 'libexpat1' was not found
0.497 E: Version '2.4.7-1' for 'libexpat1-dev' was not found

(The full job output is here.) Do you have any idea why it can't find the older files?

The apt cache has been purged by the preceeding line, so you'll need to add an apt-get update && to the start of the RUN command I think?

[ptheywood]

@bdevans I've submitted a PR which adds the missing apt-get update &&. Running this locally the docker build succeeded.

Chaste/chaste-docker#18

[bdevans]

Good spot - thanks!

[ptheywood]

The upstream VTK issue has been marked as closed on 2024-04-19 by https://gitlab.kitware.com/vtk/vtk/-/commit/db8f9efca220c9d16a30958e179abae3379d0011

If or when this gets backported to ubuntu 22.04's repositories (and other effected distributions, including 24.04 and 23.10) then .vtu files containing <appendedData> elements should be usable once again.

[jmpf]

I've been looking at this. Specifically using Ubuntu 22.04 with the newer libexpat

ii  libexpat1:amd64                            2.4.7-1ubuntu0.3                            amd64        XML parsing C library - runtime library
ii  libexpat1-dev:amd64                        2.4.7-1ubuntu0.3                            amd64        XML parsing C library - development kit

• Failure of TestVtkMeshReader::TestFilesOpen can be fixed by reconstructing mesh/test/data/cube_2mm_12_elements.vtu via TestXmlMeshWriters::TestBasicVtkMeshWriter
• Failure TestVtkMeshReader::TestGetNextElementData can be fixed by removing mesh/test/data/sids.vtu and the lines that read it. The required functionality was deleted in de0c50b

The next failure poses a problem. In reconstructing heart_decimation.vtu we should run TestXmlMeshWriters::TestVtkMeshWriterWithData(). However, with the newer libexpat this test writes a file that it cannot immediately read back!

[ptheywood]

Current plan decided at the Sheffield in person event is to hold off on this until the end of July, hoping that this will be backported upstream to the Ubuntu 22.04 repository and fixed in the 24.04 repository by then.
CI is in place to add a comment on this PR if/when 22.04 is fixed.