Maintain consistent element order in exports

[jonbartels]

Is your feature request related to a problem? Please describe.
I am looking at diffs of exports from MC and get a lot of noise from things like:

• code template library dependency lists
• code template library context lists

Describe your use case
Sorting these exportable items would provide for more consistent diffs when comparing exports. The current pattern is noisy.

Describe the solution you'd like
I think these values are a Set ensuring they're implemented as a sortable Set implementation like TreeSet and sorting them before export would work.

Describe alternatives you've considered
XML-aware diffing is possible but I haven't found a good tool to do it yet.

Additional context
I intend to implement a PR for this change. It might be a bit until I get around to it. My feelings won't be hurt if someone else beats me to it though!

I expect this problem exists in other areas of MC beyond the examples I spotted. I am not sure how to find those.

I observed this on MC 3.9.1. I need to check if it is an issue on 3.12 or in the main development branch.

[sproket]

I agree. This is an issue when using mirth exports under version control.

[jonbartels]

#3704 Same issue report made by NG staff and commented with a similar solution to what I proposed.

[ChristopherSchultz]

Yes, please. Using channel definitions with revision-control means this is a problem which turns small changes into huge-looking ones.

[jonbartels]

The problem is further visible from NextGens own Channel History extension. The extension is well worth the price of the core extension bundle but we can see here that diffing two code templates is a pain in the butt. The inconsistent element order makes the diffs funny:

[tonygermano]

Any class that ends up in a server configuration export should declare fields as SortedSet instead of Set and SortedMap instead of Map.

[jonbartels]

@rogin took a crack at this in a fork - development...rogin:connect:issue-5004

For anyone watching this issue - Can you please do a code review or test a local build from Rogins fork? He asked for some help with testing in Slack.

[jonbartels]

I reviewed @rogin fork. As a customer paying for the Channel History plugin and also integrating my Mirth exports into source control, I ask and encourage NextGen to incorporate rogins work. His work solves an annoyance for BOTH community and premium users.

A casual hand test seems to show the changes working as expected. The diffs are a lot tighter:

➜ diff 2023-05-20\ Mirth\ Backup.xml 2023-05-20\ Mirth\ Backup.2.xml          
2c2
<   <date>2023-05-20 13:08:37</date>
---
>   <date>2023-05-20 13:09:22</date>
1191c1191
<       <revision>1</revision>
---
>       <revision>2</revision>
1193c1193
<         <time>1684602500152</time>
---
>         <time>1684602558403</time>
1199d1198
<         <string>ae998ba0-dfa4-4408-a1c4-5dc14d52a088</string>
1203c1202,1204
<       <disabledChannelIds/>
---
>       <disabledChannelIds>
>         <string>ae998ba0-dfa4-4408-a1c4-5dc14d52a088</string>
>       </disabledChannelIds>
1208c1209
<           <revision>1</revision>
---
>           <revision>2</revision>
1210c1211
<             <time>1684602500181</time>
---
>             <time>1684602558433</time>
1214a1216
>               <contextType>CHANNEL_DEPLOY</contextType>
1215a1218
>               <contextType>CHANNEL_UNDEPLOY</contextType>
1221a1225
>               <contextType>CHANNEL_ATTACHMENT</contextType> 

A sampling of previously problematic areas shows ordering:

      <enabledChannelIds>
        <string>ae998ba0-dfa4-4408-a1c4-5dc14d52a088</string>
        <string>bf1f325b-e055-46d9-910f-2ae2e3bea41a</string>
        <string>23ae9d81-df20-4bf5-a9b5-3132ccde7c8a</string>
      </enabledChannelIds>

I reviewed #5004 and #3704. @rogin appears to have updated all previously known areas. I have encouraged @rogin to submit a PR.

[pujux]

This issue is a real pain for anyone trying to version control their Mirth code.

I came up with this XSLT file and a bash script that runs the formatting:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:strip-space elements="*"/>
    <xsl:output method="xml" indent="yes"/>

    <xsl:template match="node()|@*">
        <xsl:copy>
            <xsl:apply-templates select="@*|node()"/>
        </xsl:copy>
    </xsl:template>

    <xsl:template match="revision">
        <revision>1</revision>
    </xsl:template>

    <xsl:template match="lastModified">
        <xsl:copy>
            <time>1688040000000</time>
            <timezone>UTC</timezone>
        </xsl:copy>
    </xsl:template>

    <xsl:template match="delegate">
        <xsl:copy>
            <xsl:apply-templates>
                <xsl:sort select="text()"/>
            </xsl:apply-templates>
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

#!/bin/bash

# Applies the sort-xml.xslt file to all .xml files in the import directory

exports=$(find ../import -name '*.xml')

echo "Formatting exports from Mirth"

for export in $exports; do
    echo "- Formatting $export"
    xsltproc -o "$export" ./format-xml.xslt "$export"
done

echo "Finished"

So this is what I'm currently using to get rid of the unnecessary changes returned when exporting from Mirth. Ideally, I wouldn't need to do all of this 😉

[daniel-ls-howard]

@pujux It would also be possible to use this idea within mirthsync itself. I created an issue over there for this.

SagaHealthcareIT/mirthsync#66

[daniel-ls-howard]

If anyone is using @pujux solution you might want to also add:

`
<xsl:template match="enabledChannelIds">
xsl:copy
xsl:apply-templates
<xsl:sort select="text()"/>
</xsl:apply-templates>
</xsl:copy>
</xsl:template>

<xsl:template match="disabledChannelIds">
xsl:copy
xsl:apply-templates
<xsl:sort select="text()"/>
</xsl:apply-templates>
</xsl:copy>
</xsl:template>
`

[anibal2j]

I want to add something else to this. The fix here would help in this use case explained below:

I have two Mirth instances on two different virtual machines. Let’s call them, master and slave. Slave was made out of a snapshot of Master a couple of weeks ago (so it's mostly up to date with Master). The idea is to have the same configuration (template code, channels, filters, transformers, configuration map, etc.) on both machines [I don’t need the data itself inside the channels, just the configuration and code].

So, I go to Master and do a Backup, I go to Slave and I do an import of the backup file from Master. I then do an export from Slave. The file exported from Slave is different from the file exported from Master.

As you'd expect the export from Slave should match the export from Master (except maybe for some timestamp here and there). Right now I actually get files that are of different size (!). I think the export/import workflow works, but I find that if you can't compare the two files and have them be the same, you don't really know. BTW, I'm automating the above with a Python script using the REST API for Mirth (/configuration endpoint). That's how I found the issue.

[pujux]

@anibal2j you can use the XSLT document I provided to format your XML after exporting so you can really compare them 😄