SSL configuration does not enable metrics being exposed through HTTPS #688
Comments
|
|
|
Thanks @dhoard. |
|
@csrrmrvll All the underlying pieces are in place. @fstab is working on configuration changes that would allow |
|
Great news, I will be waiting for this feature to be released. Thanks a lot |
|
@fstab - is the SSL configuration to allow jmx_exporter to support both SSL/TLS termination and basic authentication being scoped ? We are keeping fingers crossed for this feature |
|
Here's a status update: Here's a bit of brainstorming what tests are missing: At this point coding is not the bottleneck, but researching how to configure different SSL scenarios can be surprisingly time consuming. It took me several hours to get the initial test running because I was fighting a "Remote host terminated the handshake" Exception. If anyone wants to help out: Please set up some of the scenarios above using the JAR files from the
|
|
Ran JMX exporter as Java agent, as mentioned in readme with 'new-config' branch code with config:ssl:true Getting errorCaused by: io.prometheus.jmx.Config$ConfigException: Configuration error in collector: Cannot set ssl=true without specifying hostPort or jmxUrl Tried with below config as well, getting err with config:ssl:true Error:Jul 08, 2022 8:58:15 AM io.prometheus.jmx.JmxCollector exitOnConfigError what i am missing? i am trying to export metrics as ssl enabled. |
|
@fstab / anyone working on the tests for Keytool / keystore instructions Java < 11 requires a JKS keystore. For Java 11+, a JKS or PKCS12 keystore can be used (PKCS12 is the recommendation for Oracle Java 11+) Create a self-signed JKS keystore, validity = 3652 days (10 years), password = Convert a JKS keystore to PKCS12 keystore (recommended for Java 11+) Test application command line configuration for full JMX security (SSL, authentication, authorization) Create a JMX access file Example content: Create a JMX password file Example content: Note: Both Start the test application with full JMX security (SSL, authentication, authorization) Start the Note: The values above and In the examples above, I have attempted to match these with Changing the test application arguments allows you to disable JMX SSL, JMX authentication. |
|
@dassan1505 ... ... appears to be the old configuration format used to configure JMX SSL. When running as a Java agent, you don't need them. |
|
@csrrmrvll @srinirama @palashbhowmick I have been working on the initial code to support HTTPS and HTTP Basic authentication. A potential pre-release version of the code can be found at...
Due to the vast number of changes, it will take time to be merged and released. |
|
Attached is a zip of potential pre-release version based on (REMOVED - proper basic authentication and SSL support is in progress) Additional configuration (optional) |
|
@dhoard appreciate your work! any ideas when it'll be merged? Just eagerly waiting to make use of tls and basic auth :) |
|
The feature branch has SIGNIFICANT project/code changes to review. I'm not a maintainer so can't speak to when these features would be released. I would suggest forking my repository/branch, build it, and test it out in your environment. Early feedback for such significant changes is much appreciated! I implemented the features along with an extensive integration test suite (~19k total test method executions over 43 Docker containers/Java versions)... because I required the functionality. |
|
@dhoard I just got around to messing with the jar you linked from a month ago. (the 17.3-SNAPSHOT.jar). For context, I'm using the jmx exporter as a java agent for exporting JMX metrics from Kafka. This is how it's normally setup for us. We export this environment variable for when Kafka starts to run this as a java agent. Our config.yaml file is usually just empty and works fine, exposing metrics on that port. However, I just replaced it with yours like so And Kafka refuses to start. This is the error I was able to capture regarding this. This happens whether i have something in my config.yaml, or if I try to enable basicAuth. The strange thing is, when i run it standalone like so It runs just fine. And even basic auth seems to work great! Just doesn't capture our Kafka JMX metrics unfortunately. so i guess something weird happens when trying to run the javaagent with that jar through a KAFKA_OPTS env variable. Does that error give you any idea what it could be? Java Version we're running: |
|
Actually nvm @dhoard , i am able to get past that error after building with your latest code and Kafka now runs uninterrupted without that error. Unfortunately now grabbing those metrics on that port takes a much longer time, with or without basic auth mentioned in the config.yaml. When I do have basic auth in the config, it does eventually load, but never actually prompts me for a username nor password. Config.yaml |
|
@euthuppan thanks for testing! Can you share your Kafka version and exact exporter configuration YAML? Typically, in Kafka monitoring scenarios, exporter configuration key Regarding BASIC auth, I'll review the code. All integration tests pass as expected. Does the agent output show that authentication is enabled? |
|
No problem @dhoard , thank you for having the initiative to make these features supported! And sure, I'm running Kafka version 3.2.0. config.yaml i'm using: I tried checking for agent output in our logs but I guess either they either aren't enabled or I'm just not sure where to check for them. I looked in our existing logs and don't see any related to the agent unfortunately. 😕 |
|
@euthuppan if you are running the application via systemd, the systemd journal should contain some standard information... I wrote some more integration tests, which appear to work correctly. Given If the endpoint is called without authentication/invalid authentication credentials, then an HTTP If the endpoint is called with valid authentication credentials and you are within the If the endpoint is called with valid authentication credentials and you are past the My time has been constrained but will try to test Kafka specifically as soon as possible. |
|
@euthuppan I performed actual testing against a Kafka 3.2.x broker and it's working as expected (described above.) Configuration YAML: Testing time via curl... Official 0.17.2 release: Build from my branch (SSL and BASIC authentication disabled): Build from my branch (with SSL and BASIC authentication enabled): |
|
@dhoard thanks a lot for setting up the test environment to test Kafka! I made my config.yaml now look just like this: And i still get similar behavior on my side. Are you setting up the JMX exporter by adding the javaagent and jar as an environment variable for the Kafka application to make use of like this? If not, how are you setting it up to scrape Kafka metrics? If there's an alternative that seems to be working for you, I'd be down to try. Still when I use this jar and specify basic auth in the config.yaml as shown above, after I hit the endpoint it does not prompt be to provide basic auth. Instead after about a minute it gives me a 200 OK and spits out the JMX metrics without providing credentials. I'm guessing if you pass in the -javaagent flag when starting kafka, you might see similar behavior to me, unless there's something dumb i could be overlooking? |
|
Here's some extra info, when i check the process running the kafka application, these are all the args i pass in. Anything here i might be missing? |
|
Looking at your classpath, you have multiple instances of the exporter in your classpath, which seems incorrect to me. ... along with the |
True, it's a little confusing because that section is just listing all the jar files in that directory, but you'll notice that only the 17.3 one is passed into the -javaagent field. Ive tested swapping that section with 17.2 and 16.1 exporter jars and it changes behavior as expected. |
|
I'm running my server via systemd Are you running via systemd? If so can you run... before starting the service. If you're not running via systemd, then the output (System.out) would be in whatever file you capture the output. Example output: |
|
@dhoard Great news! I finally figured it out. Your original hunch was spot on, it was the extra jmx exporter lib versions laying around in the classpath that seemed to make the difference. After removing those older jars, everything plays nice! Maybe it caused the jmx exporter logic to get conflated with the other libs.. not sure. I guess I assumed just placing the new jar in the -javaagent field would be sufficient, but seems that was the wrong assumption. Now I finally get it prompting me with Basic Auth when i hit the endpoint in my browser! 🙌 Btw I'm running it in a container, and i am controlling starting and stopping that container through systemd so journalctl only shows the container state. But i was able to start seeing the JavaAgent output using |
|
@euthuppan great news! |
|
Thanks a lot for working on SSL support. Just a quick status update from a maintainer: My current priority is on a new data model for |
|
@dhoard I've been using your version of the jmx_exporter from back in February (only in a couple QA kafka clusters) and it works pretty great for TLS and basic auth support. It's fantastic! I just had a couple things i noticed while using this on a real QA kafka cluster the last couple months.
|
|
@euthuppan I'm not aware of any changes that would cause the slower scrape or the unresponsiveness (It might have been something in 0.17.3 SNAPSHOT codebase I used - I no longer have the branch.) I'm working on proper basic authentication functionality. Once completed, I'll start working on proper SSL configuration. Again, I would advise you to NOT use the code in production. |
|
gotcha @dhoard so you've essentially started rewriting it? and yep note taken. |
|
@euthuppan correct. It will a proper feature. |
|
@euthuppan This is targeted for the next (post 0.18.0) release |
|
Resolved in release 0.19.0. |
I am trying to expose metrics via HTTPS or SSL on TCP and being scraped from a central Prometheus as there are firewall rules in place to block non-cyphered connections in our network. I have configured jmx_exporter as documented, but I can only access metrics over HTTP. This is the config:
version
jmx_prometheus_javaagent-0.16.1
config.yaml:
ssl: true
lowercaseOutputLabelNames: false
lowercaseOutputName: false
startDelaySeconds: 0
JAVA_OPTS
-javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent-0.16.1=10001:/etc/jmx-exporter/jmx_prometheus_javaagent.yaml -Djavax.net.ssl.keyStore=/etc/jmx-exporter/keystore -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.ssl.trustStore=/etc/jmx-exporter/truststore -Djavax.net.ssl.trustStorePassword=changeit
What am I missing?
The text was updated successfully, but these errors were encountered: