OutOfMemory occurs on many OdfTable creations.

[juniair]

I'm using this library to batch create reports on a system I'm developing. Meanwhile, an OOM issue occurred while writing the report. The report that OOM occurred was a report that required more than 1 billion tables.

This problem occurs in the following situations:

Case 1. OOM occurs when more than 1 billion tables are created for one document
Case 2. OOM occurs when storing more than 1 billion tables in multiple documents

When I checked the JVM dump, it was confirmed that mTableRepository in OdfTable uses about 90% of memory.

When I checked the source code to figure out the cause of this problem, mTableRepository was designated as static.

Currently, the system I am using is compatible up to JDK 1.8, so only OdfToolKit version 0.9.0 can be used.
What should I do to solve this problem?

[rudiger1986]

Hi,

I have the same problem. The problem seems to be in the static map mTableRepository, where it is storing OdfTable objects that are not being released. Moreover, in my case I am also creating OdfTableRow objects that in the end are not released because of the references to the OdfTable that are stored in mTableRepository.

I am using jdk 11 and the latest version 0.10.0.

Note: The problem resides in OdfTable, when you call the getInstance method and store it in the static map mTableRepository .

static IdentityHashMap<TableTableElement, OdfTable> mTableRepository =
      new IdentityHashMap<TableTableElement, OdfTable>();

public static synchronized OdfTable getInstance(TableTableElement odfElement) {
    if (mTableRepository.containsKey(odfElement)) {
      return mTableRepository.get(odfElement);
    } else {
      OdfTable newTable = new OdfTable(odfElement);
      mTableRepository.put(odfElement, newTable);
      return newTable;
    }
  }

Any solution for this blocking?

Thanks

[svanteschubert]

Thank you all for your feedback, I have discussed this problem with the co-maintainer Michael Stahl (@mistmist) today.

First, we believe this problem is related to #173 and worth fixing for our next release.

As @rudiger1986 has pointed out the static mTableRepository of OdfTable seems to be the core of the problem, as this static Map is only being filled but never emptied:
see https://github.com/tdf/odftoolkit/blob/master/odfdom/src/main/java/org/odftoolkit/odfdom/doc/table/OdfTable.java#L113

There are at least two ways to fix this problem:

1. The easiest solution - if working as expected - is to exchange the IdentityHashMap with a WeakHashMap.
   Our assumption is that the key - the ODF XML element TableTableElement being a weak reference in a WeakHashMap - is only weakly reachable if no other strong references exist to this DOM TableTableElement object like from the XML DOM parent. The DOM itself will be released (indirectly) by the close() method of the [OdfDocument] (https://github.com/tdf/odftoolkit/blob/master/odfdom/src/main/java/org/odftoolkit/odfdom/doc/OdfDocument.java#L706)
   or on one of its subclasses.
   The difference is that the currently used IdentityHashMap always kept a strong reference to the XML DOM as key and the DOM was never collected by GC.
2. The second solution, which costs a bit more programming effort is to transform the static collection at the OdfTable to a non-static instance Map at the DOM instance of the OdfElement provided as a parameter. The ODFDOM DOM class is called OdfFileDom and can be received via OdfFileDom fileDom = (OdfFileDom) odfElement.getOwnerDocument();.
   The collection of the OdfFileDom instance would be resolved, when the close() function will be called at OdfDocument, calling its super.close() on class OdfSchemaDocument, which refers to the ODF XML DOM instances.
   We could keep the API and move behind the scenes the Map.
   But when following the second approach, another topic worth mentioning is the synchronized approach of the OdfTable Map.
   While earlier the Map would have to deal with the work on multiple ODF documents and their XML DOMs now it would be only on one XML DOM of one ODF Document. As all the other functions to work on a single DOM are not synchronized Michael and I believe we could safely remove the synchronized and exchange the Vector implementation against ArrayList (dropping the setSize functionality that does not exist explicitly in ArrayList but only in Vector).

Would you agree on the above observations?
And if so, would anybody of you - having the appropriate test environment at hand - be so kind to test one give one of the above solutions a try?

PS: Remember the three-layer design concept of ODF / ODFDOM:

1. The PKG package layer could - in theory - be reused for any zipped format having the ODF manifest, encryption and signature. EBUP was initially reusing ODF 1.0 Package until they invented their own signature/encryption due to miscommunication (no one was aware of the work of the other) between the Technical Committees.
2. The DOM layer is given by the ODF grammar (the schema). It has ODF XML as an implementation detail, but in theory, a ODF document can also be expressed as a list of ODF changes (e.g. in JSON). Only by switching from the final state to state changes concept the core question of collaboration on a single document with multiple users ("what have you changed?") can be answered! ;-)
3. The highest layer is the semantic layer we coined the DOC layer, which might be in the end free from implementation detail (as XML)

• The OdfFileDom from the org.odftoolkit.odfdom.pkg java package is in the package layer representing any XML within a zipped Package.
• The OdfSchemaDocument from the org.odftoolkit.odfdom.dom java package represents the concept of a document in the ODF XML Schema layer being a directory with a MimeType in the Manifest and having several XML files: content.xml, styles.xml, meta.xml and settings.xml.
• The OdfDocument from the org.odftoolkit.odfdom.doc java package represents the semantical concept of a document (here still inherited from the XML implementation details) but in general could be independent of it.

[svanteschubert]

One slightly design improvement for the second solution: instead of using an OdfFileDom instance as the new holder of the mTableRepository (which does not know anything about ODF XML in the package layer), we might want to add a new common parent class for

• OdfContentDom
• OdfStylesDom

perhaps named OdfContentContainingDom at the org.odftoolkit.odfdom.dom java package being aware of ODF XML.

Remember: OdfStylesDom contains the content of Header & Footer, which may contain a full document (aside from page breaks)

[svanteschubert]

@juniair We have uploaded a patch, could you please pull, build and test your scenario again, please?
Thanks in advance!
Svante