log4j vulnerablity - is Mirth affected?

[rangercej]

News is breaking that there's a remote code execution vulnerability in log4j. I'm not sure what version of log4j ships with Mirth, but there are mitigations.

Questions are:

• Does Mirth 3.4.x or higher use one of the vulnerable log4j libraries?
• Are we able to use the formatMsgNoLookups propety in one of the .properties files to mitigate against it without breaking logging?

More details: https://www.lunasec.io/docs/blog/log4j-zero-day/

[edgarecayce]

Came here searching for same thing - just looking at my install of 3.9.1 i find 8 instances of log4j-1.2.16.jar (and no other versions) - but I have no idea how to get Mirth to use a newer one.

[jhowe-uw]

https://nvd.nist.gov/vuln/detail/CVE-2019-17571

However, it's still in a "re-analysis" phase. Reading through all the apache threads, 1.2.17 was EOL when they started on Log4J 2. Some threads say this socket exploit was fixed in Log4J >= 2.8.2

[robertojnior]

Any updates on this?

[eric-hf]

@gbortis, @narupley, @bmoenng, @pladesma, @leilanimng, @cturczynskyj, @Fdawgs, @igorazevedo, @pattersp, @twest-mirthconnect: with these older log4j versions baked in with Mirth, is there a way we can upgrade-in-place or otherwise mitigate this issue?

[edgarecayce]

I am seeing people say that the 1.xx versions of log4j are not vulnerable, only the 2.x ones < 2.15. Not 100% sure if this is accurate

[eric-hf]

Apache only said that versions <=2.14.1 were impacted. The "versions >=2.10" reference I see is just whether one workaround setting is available.

The Mirth tar balls all still use %m in the log4j message formats, so I think you'd still be at risk of injection since the wildcard would fill in depending on what the logged message said.

[rangercej]

According to this comment on the fix PR by @remkop, v1 is sounding possibly vulnerable (they keep changing their mind on it): apache/logging-log4j2#608 (comment)

@narupley - any thoughts?

[narupley]

There is a separate JMSAppender that does support JNDI for looking up the binding / connection factory, but those are just strings, and I don't know if any actual deserialization would be happening in that case.

In any case, we do not use the JMSAppender class for logging.

[narupley]

The founder of log4j said this on twitter:

Log4j 1.x does not offer a look up mechanism. Log4j 1.x sends an event encapsulating a string message to a JMS server. That is it. The attacker can supply whatever string he chooses but it remains a String. So not the same. At all.

https://twitter.com/ceki/status/1469449618316533762

[narupley]

Mirth Connect still uses log4j 1.2.16, and doesn't include log4j 2.x. I tested logging out the exploit string described in the vulnerability, with a network capture going at the same time, and confirmed that the JNDI connection is not being made. I did that testing in older versions of Java like 8u60 as well.

In fact, that JndiLookup class isn't even present at all in the log4j 1.x JAR. That looks to have been added in 2.x: https://logging.apache.org/log4j/2.x/manual/lookups.html#JndiLookup

So as far as I can tell, MC is not affected by this CVE, unless you are explicitly including log4j 2.x as a custom library. And even if you are, you still won't be vulnerable as long as you're using one of these Java versions or newer:

• > 6u211
• > 7u201
• > 8u191
• > 11.0.1
  If you are including log4j 2.x as a custom library and are using an older version of Java and can't easily upgrade, then follow the mitigation instructions and set the log4j2.formatMsgNoLookups system property to true, as well as the com.sun.jndi.rmi.object.trustURLCodebase and com.sun.jndi.cosnaming.object.trustURLCodebase system properties to false.

As a side note we also have an item on our roadmap to upgrade log4j from 1.x to 2.x! It was just dumb luck I guess that we had not gotten around to doing that yet.

[jtc42]

It's worth noting here though that in the NHS Digital annoucnement of this vulnerability they do say:

Note: Version 1 of the Log4j library is no longer supported and is affected by multiple security vulnerabilities. Developers should migrate to the latest version of Log4j 2.

While it's great that Mirth seems unaffected due to using log4j 1.x, I'm not sure if that's all that comforting... Hopefully the vulnerabilities they mention have been worked around by Mirth.

[rangercej]

log4j v1 has received its own CVE now to remove confusion (CVE-2021-4104). Appears it's only a problem if you use JMSAppender, but suggest folk review and keep a watch on:

Sources:

• https://lists.apache.org/thread/0x4zvtq92yggdgvwfgsftqrj4xx5w0nx
• https://access.redhat.com/security/cve/CVE-2021-4104

[vincentvent]

Hi @narupley ,
We have been using Mirth for years and we love it. Thank you!
I am hitting a snag for a critical go-live where I am being asked what the reasoning is for tolerating these vulnerabilities and what the plan is for remediation in the long term. I would like to respond something like:

The Log4j vulnerability is not part of Mirth which is using Log4J 1.2 and confirmed that the JNDI connection is not being made.
Additionally, other vulnerabilities that might be part of the Log4J version used cannot be exploited if the configured ports are blocked by the firewall (default TCP 8080 + 8443).

Update to the Log4J library is planned for Q3 2022 for Mirth Version 4.0.X

Thanks again!

[ChristopherSchultz]

Hi @narupley , We have been using Mirth for years and we love it. Thank you! I am hitting a snag for a critical go-live where I am being asked what the reasoning is for tolerating these vulnerabilities and what the plan is for remediation in the long term. I would like to respond something like:

Was the existing reply not sufficient?

[vincentvent]

Sadly not. We need in a single package:

• Why this is not an issue
• Steps to mitigate this and other issues from being exploited
• Commitment to update (target dates would help here)

Thanks!

[rangercej]

As a heads-up for folk as well, if you have pulled log4j v2 in as an external, Java versions later than (say) 8u191 may be affected through different JNDI vectors. I admit I don't know enough Java to know how realistic this is though, but have been signposted to https://www.veracode.com/blog/research/exploiting-jndi-injections-java

[jtc42]

I know I posted this in a thread above too, but it's worth noting that NHS Digital specifically discourage use of log4j 1.X, so anyone in the UK at least using Mirth is likely getting caught out by that.

[kthr]

The problem with log4j version 1 is also that it had its end of life on August 5, 2015. So nobody knows what other security vulnerabilities are there, and nobody looks for them either. As mirth is a communication server which handles clinical data I think updating log4j is rather critical (not saying that the other updates are not important too).

I did a scan of nextgenhealthcare/connect:3.12 with trivy that shows 10 critical vulnerabilities. I don't know though which of these would really be applicable.

[psapozh]

Is there any update on CVE-2019-17571 issue mentioned by @tippenring. Is mirth affected by this?

[pcdewolff]

so #4487 should get a (way) higher priority than it has now. Actually with a solution included

[ChristopherSchultz]

People are definitely still looking for vulnerabilities in them, and they are still being reported. It's just that the project isn't releasing any new versions to address these vulnerabilities. For the most part, these security issues are in unusual uses of the library and do not affect "typical" uses. For example, log4j 1.x includes a socket server component which has some vulnerabilities. But literally nobody uses that component for everyday logging, so the vulnerability may be present (e.g. in MC Server up to and including 3.12) but it is not exploitable in any way. (I'm not suggesting that I know this for a fact about MC Server, but I would be very surprised to find out that MC Server is using the socket server from log4j for some reason.)

[edmundgr]

@ChristopherSchultz these are good points but they do not matter to our customers some of which are in the DoD. The safest and only acceptable course is to upgrade log4j to a 2.x version that is not vulnerable. This is especially true given the high level attention this vulnerability has received. Also, some log4j vulnerability "mitigations" have later proven to be ineffective. Is there a reason not to do this upgrade of log4j?

[tonygermano]

@edmundgr log4j 2.x has released 3 security patches this month, while log4j 1.x has no known vulnerabilities that affect mirth. I'd say that's a great reason to not force an upgrade immediately.

[willcey]

As of yesterday there is a new vulnerability which affects log4j Version 1.2
https://nvd.nist.gov/vuln/detail/CVE-2021-4104
How could we react on that? Is Mirth even affected by this issue?

[evulhotdog]

deserialization of untrusted data when the attacker has write access to the Log4j configuration

Sounds like the system is already exposed if someone has write access to the log4j configuration. Even if it is affected, doesn't seem like a very high risk.

Edit: See https://security.snyk.io/vuln/SNYK-JAVA-LOG4J-2316893 for how to take advantage of it.

[robertojnior]

I think this comment (#4892 (reply in thread)) is related to this CVE.

[Chewwy420]

Is there any official works from Mirth about this or a fix.. We are getting hammered by our customers about this issue...

[tonygermano]

@edmundgr FYI, @narupley (the guy with a Maintainer tag) is the only NextGen employee that has responded to this post. The rest of us are all users like you.

Your premise is wrong that you aren't getting a secure product. The version of log4j that mirth is currently using has so far been shown to be more secure than versions that were released even after this discussion post originated. When you are doing a risk analysis you need to determine not only if there are vulnerabilities reported for a particular piece of software you are using, but also whether or not you are using the software in a way that the vulnerability applies. In mirth's case, none of the log4j 1.x vulnerabilities apply unless the user has added in customizations which do not ship with mirth to activate these rarely used vulnerable features. Additionally, these rarely used vulnerable features are not compatible with log4j 2.x, so either way it would be up to the user, not mirth, to rectify the problem.

[tonygermano]

@edmundgr So the question should not be whether you are on the newest version of a piece of software that recently patched several vulnerabilities that affected many users and the dust still has not settled. The question should be whether you are on a secure version of the piece of software. The answer to that is "Yes, we are using a well-established version that has no known vulnerabilities for our use case."

[twest-mirthconnect]

Hey Gents,

If you missed the webinar earlier this month, we reviewed and documented why Mirth Connect is NOT vulnerable with @narupley .

Here is where we are today for Mirth Connect:

• Security updates are defined and provided by our security champion, Nick Rupley. After review, there is no problems for Mirth Connect. Yesterday, we reported to the community and demonstrated why Mirth is not at risk. Nick did an excellent job explaining the vulnerability and that we are ok.
• Updates in AppSec, Tech Debt, and associated maintenance are defined and prioritized by our Architecture team. I rely on our team to curate and prioritize this list of work which is included in each release.
• Log4Shell is not a high priority to date but has been included and will be addressed in FY 2023 (3.13 – 3.16)
• Dates for releases are difficult to lock in without due diligence with the team, business priorities, and justification.
  

@pacmano1 mentioned that we prioritize paying customer requests over the open-source community... this is both right and wrong. The roadmap has both. The more folks jump into mock-ups, requests, and conversations the more the team and I can work through how to respond and provide the best product for everyone.

The easiest thing to do is get involved here, slack, setup a call with me on Friday mornings, or send what you are interested in through the ideas portal.

[ChristopherSchultz]

Quick question for @twest-mirthconnect and @narupley - the above message says the log4j upgrade would be scheduled for FY 2023 [sic] and 3.13 - 3.16, but NG has now released 4.0. Can we get an update on the expected schedule for this library migration? Was the "FY 2023" a typo?

[twest-mirthconnect]

ah yes, we moved the numbering due to CURES. The log4j update will happen in the next release 4.1. Thanks for asking Chris.

[tkz91]

Considering the vulnerabilities that the version of Log4J that Mirth Connect uses has, please upgrade to a newer version so we can recommend upgrading to a version of Mirth that will address these concerns.

[jonbartels]

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5645 and the relevant patch apache/logging-log4j2@5dcc192

This is a risk in the Log4J org.apache.logging.log4j.core.net.server.AbstractSocketServer and its implementing classes which are not the same as java.net.ServerSocket

[tonygermano]

The vulnerable Socket Server is to listen for log events coming from remote log4j processes not running in the same JVM. I'm pretty sure that it doesn't even exist in log4j 2.x, so if you are vulnerable to this log4j 1.x risk because of something you configured, your code will break when nextgen upgrades to log4j 2.16+.

[ChristopherSchultz]

@jonbartels is correct. There are many classes in many packages and libraries in Java called "server" and "socket". Just because one of them has a similar name doesn't mean it's used anywhere. The log4j SocketServer class is literally a server process: its a class with a big main method that binds to a socket and listens for incoming connections so that it can act as a log-message receiver. Nobody uses it. If you are going to log to the network, you use syslog and let the adults handle the situation.

[tonygermano]

I would revise the original comment to say, "considering the vulnerabilities that the version of Log4J that Mirth Connect uses has, please stay on this version until the log4j 2.x issues have been worked out. There is virtually zero risk with remaining on the currently used version of log4j."

[ChristopherSchultz]

The real problem here with everyone complaining is that of false equivalence. Users like @edmundgr and others are saying "the latest version of log4j is 2.17.0, and that is the only safe version." That is true, but only when phrased like this "the latest version of log4j 2 is 2.17.0, and that is the only safe version of log4j 2."

You should really think of log4j 1.x as a separate product which has no attachment whatsoever to log4j 2. Imagine if all the developers at Microsoft who worked on Windows OS left Microsoft, abandoning Windows and started a new OS called FooOS, and a few years down the line, FooOS ended up having a bunch of security vulnerabilities that never existed in Windows. Would you demand that all Windows users upgrade to the latest FooOS?

Sound ridiculous, right? Well, that's essentially what you are asking for, here.

Log4j reached "end of life" which means that the developers no longer release new versions of it. Log4j 2 is a completely different product. Yes, there are a lot of similarities between the two products and yes, there are a lot of the same developers working on it and yes, they still even use the same mailing lists to communicate.

But while log4j 1 has reached end-of-life that doesn't mean it's no longer supported. Go ask a question on the mailing list, and it will be answered. Yes, there are public CVEs reported and no, they won't be fixed (because they have stopped issuing releases), but the product can be made safe.

I had a client who this week required me to remove JMSAppender.class from the log4j-1.2.17.jar file for them, even though it isn't in use. The only one who has the power to enable the JMSAppender is them. But, whatever, I guess: they are the client. I wonder how up-to-date their security people are, there, as they didn't require me to remove ServerSocket.class from the JAR file.

But that's the point: if you perform an accurate risk assessment, there is no danger, here: the product, in the way it is used, is not vulnerable. If you are super scared then you can actually neuter the product in such a way that it's not even possible to use it in a dangerous way. Note that the changes log4j 2 made in recent days are exactly the same. First, they disabled the enablement of that feature by default, then they removed it entirely. It's equivalent to not using the JMSAppender (via configuration), then removing the class.

log4j (nor any other Apache Software Foundation project) never guarantees the safety and security of a single release (and you should be suspicious of anyone who does, regardless of the software product). There could still be things lurking inside log4j 2, or log4j 1, or your JRE, or your OS. If something Bad is discovered in the Windows Subsystem for Linux, but you don't use WSL, are you in a huge hurry to apply those patches?

I know I'm wasting my breath. Risk analysis apparently isn't your thing; you just want to show someone that Y > X and therefore everyone is 100% safe. If you are that superficial about it, just re-name the JAR file from log4j-1.2.17.jar to log4j-2.safe.jar and call it a day.

[kthr]

@JoeFox82 proposal

There would be an easy way to upgrade to 2.16 because Log4j Version 2 has a bridge API in its package called log4j-1.2-api-2.16.0.jar so that you can replace the version 1.2.x without changing your code.

sounds quite good and probably doable quite fast. If I'm not wrong we would only have to delete the log4j jars here:

./donkey/setup/lib/log4j-1.2.16.jar
./donkey/lib/log4j-1.2.16.jar
./server/lib/log4j-1.2.16.jar
./generator/lib/log4j-1.2.16.jar
./manager/lib/log4j-1.2.16.jar
./command/lib/log4j-1.2.16.jar
./client/lib/log4j-1.2.16.jar

and replace them with log4j-1.2-api-2.16.0.jar and log4j-core-2.16.0.jar.

Currently trying some compiling (unsuccessful so far)

[JoeFox82]

A user in Slack also found that there are calls to dynamically configure an appender at

connect/server/src/com/mirth/connect/plugins/serverlog/ServerLogProvider.java

Line 48 in e66d112

patternLayout.activateOptions();

. This is not supported in the Log4J 1.x -> 2.x bridge.
I haven't had time to experiment but some thoughts:

1. Can the appender be added to log4.properties manually then the initializer reduced to just the serverId and logController lines? Probably not since ArrayAppender wants an instance of ServerLogProvider
2. Removing the plugin may also be a tolerable dirty solution. IIRC this is the gizmo that shows the logs in the MC dashboard

That's correct, you'll have to compile the code because there're two things to change:

1. The bridge does not support the function patternLayout.activateOptions(); so you have to remove it (don't know side effects of removing this)
2. you have to add the bridge, the 2.16 API and the 2.16 Core to the Classpath of all sub modules like server, donkey, etc.
3. After that you have to change the class com.mirth.connect.server.launcher.MirthLauncher like this, cause there is a hard link to the Log4j jar.
   

After that you have to customize the .xml files of the ant install because you have to add the jars to your build so that they will be signed, too.
Compiling works well but I couldn't test it, yet.

[jonbartels]

After that you have to customize the .xml files of the ant install because you have to add the jars to your build so that they will be signed, too.

There are two workarounds for this:

1. If the modified files are not used in MC Client and listed in the JNLP they do not need to be signed
2. If they do have to be signed, the MC Admin Launcher provides a command flag to tolerate self-signed JARs from MC Client

[ChristopherSchultz]

Note that 2.17.0 should be used, not 2.16. Fun times at log4j headquarters.

[s2ramana]

Has anyone confirmed that the mechanism to use the 1.2 to 2.x bridge works along with the Mirth code changes? It would be good to get confirmation on this strategy as many users are asking about this.

[twest-mirthconnect]

@pladesma - Would you have time this week to answer questions from those in this string that have not closed the loop?

[twest-mirthconnect]

the log4j update is currently in development. We will have it ready for release with 4.1 the end of July.

[JCW-1087]

do you know which version of log4j will be included in 4.1?

[twest-mirthconnect]

2.17.2

[JCW-1087]

Great, thanks for your response.

[johnpfaff]

Is the 4.1 release still coming soon?

[JoeFox82]

Version 4.1.0 is already released: https://github.com/nextgenhealthcare/connect/wiki/4.1.0---What's-New

[twest-mirthconnect]

Today From: John Pfaff ***@***.***> Sent: Friday, July 29, 2022 10:06 AM To: nextgenhealthcare/connect ***@***.***> Cc: Travis West ***@***.***>; Mention ***@***.***> Subject: Re: [nextgenhealthcare/connect] log4j vulnerablity - is Mirth affected? (Discussion #4892) [EXTERNAL] Is the 4.1 release still coming soon? — Reply to this email directly, view it on GitHub<#4892 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AU5HHFWGQZEZMW6NUEO2NZ3VWPXL5ANCNFSM5JYTI37A>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>