New logs related to System Trust Manager?

[lejeunen]

Hello

We are using java buildpack 3.18 on PWS and have recently discovered new INFO log statements:

ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
ContainerTrustManagerFactory$PKIXFactory : Adding TrustManager for /etc/ssl/certs/ca-certificates.crt
org.cloudfoundry.security.FileWatcher    : Start watching /etc/ssl/certs/ca-certificates.crt
c.s.FileWatchingX509ExtendedTrustManager : Initialized TrustManager for /etc/ssl/certs/ca-certificates.crt

These seem to occur for every http request entering our app.

According to PWS support this is not related to them. Could they be related to a recent change in the Java Buildpack?

Thanks a lot for any help

Nicolas

[nebhale]

Correct. The Cloud Foundry Security provider configures both the “system” (as configured via a default or via system properties) and container-specific key stores and trust stores. This log shows that it’s configuring the system trust store as well as a trust store from the certificates managed by BOSH trusted certs.

[lejeunen]

Ok. Is it supposed to configure these stores on each incoming HTTP request, or is it an issue with CF security provider?

We are afraid this has a negative impact on response time and memory consumption. We ran into memory issues since this log started to appear.

Thanks!

[nebhale]

Interesting. A new one is created each time a connection calls TrustManagerFactory#getInstance assuming that network libraries would cache such a thing (perhaps calling for each new connection in a pool). What library are you using that calls repeatedly for it?

[lejeunen]

I searched and we did not change anything related to this in the last 4 months, so we suspect these might come from the Java buildpack itself. Something introduced quite recently.

We only observe this behaviour on our gateway component, i.e the one handling calls from the outside world. We do not observe these logs in our other micro services.

[nebhale]

The logs are definitely generated by our Security Provider; there’s no question of that. Any application staged with a recent version of the buildpack will see them whenever the system TrustStoreManager is called for. In most applications this is called for once (or rarely) when connection pools are created. If you’re seeing it repeatedly, that means that whatever code you’re using is calling for a brand new TrustStore repeatedly. It also means that before our Security Provider was added, your code was also calling for a brand new TrustStore repeatedly, it just wasn’t being logged.

Before we make any changes, I want to understand what code you’re using that causes this to happen and so I’ll repeat my request for you to tell me what library you’re currently using that repeatedly calls for a brand new TrustStore.

[imod]

We face the same issue...
In our case this seems to be caused by the way we communicate with elasticsearch. We once faced the issue that we lost the connection (TransportProtocol, not HTTP) to elasticsearch, because our hosted instance was moved to a different IP. Because of this elastic advices to disable/reconfgure DNS caching (https://www.elastic.co/guide/en/cloud/current/_dns_caching.html) like this:

java.security.Security.setProperty("networkaddress.cache.ttl" , "60")

Elastic also advises to update each client (object holding a connection) with newly resolved IPs every couple of minutes. Full story is here: https://discuss.elastic.co/t/nonodeavailableexception-with-java-transport-client/37702

[nebhale]

I'm guessing that

Elastic also advises to update each client (object holding a connection) with newly resolved IPs every couple of minutes.

is why you're seeing the request for new TrustManagers regularly (it'd be an interesting validation if the logs correlated with that). I'll be working today on doing some aggressive caching for created Trust and Key stores to minimize the number of times we create new instances.