XML parsing is very memory intensive

[jhorbulyk]

Currently, the xml2js library referenced consumes lots of memory when parsing xml. For example the metadata document for a typical MS Dynamics metadata document with a size of 10-20 MB can result in memory usage spikes that end up in the 512 MB to 1 GB range. Updating the xml library may resolve this problem.

[zubairov]

@jhorbulyk updating the library means replacing it with some other library?

[jhorbulyk]

I think the first step would be to update to the latest version of the existing library.

[ShkarupaNick]

3 SP
Estimation was made according to next actions :

1. Update version of library
2. Investigation of possible improvements of memory usage

[uaArsen]

Investigations results:

1. Updating library version will not solve a problem. We already use latest version and their git repository was last time updated 2 years ago.
2. I found another library xml-js that will partially solve this problem, heap usage will be decreased by 50% approximately.
3. elasticio/odata2openapi repository does not correspond to our libraries code of conduct. Updating accordingly is out of scope of this issue. I am proposing creating new issue and marking this as blocked.

[jhorbulyk]

• I guess I missed that ^0.4.17 in package.json evaluates to 0.4.19 in package-lock.json
• As for the libraries code of conduct, it is worth noting that someone else wrote this library and then transferred it to us. Obviously the original author did not adhere to our syle guides. In terms of getting to the requirements defined in the style guide:
  • JS style guide: This library is mostly in typescript. In general, we have opted to write vanilla JS over typescript but I'm not sure if it is pragmatic to convert this library from typescript to JS. Some other code styling rules might be minorly different from our guidlelines but if they are consistent in the library, re-styling might not be a pragmatic use of time.
  • Library log sensitivity: I don't think there are any log statments
  • Unit tests & CI: We should definitely do but is a potentially large unit of work.
  • Integration tests: N/A
  • Documentation: We should definitely do but is a potentially large unit of work.