Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to fetch secrets from Vault #4

Open
Kalyan-D opened this issue Jul 21, 2021 · 10 comments · May be fixed by #5
Open

Unable to fetch secrets from Vault #4

Kalyan-D opened this issue Jul 21, 2021 · 10 comments · May be fixed by #5

Comments

@Kalyan-D
Copy link

Hello jcustenborder ,

I have configured my confluent worker with following properties:

config.providers.vault.param.vault.token=************************
config.providers.vault.class=com.github.jcustenborder.kafka.config.vault.VaultConfigProvider
config.providers.vault.param.vault.address=http://********:8200
config.providers=vault
config.providers.vault.param.vault.login.by=Token

and started the worker in distributed mode

I have configured my source connection with the following configs

{
"name": "inventory-connector",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"tasks.max": "1",
"database.hostname": "*******",
"database.port": "3306",
"database.user": "${vault:confluent/secrets:mysqluser}",
"database.password": "${vault:confluent/secrets:mysqlpass}",
"database.server.id": "184055",
"database.server.name": "dbserver2",
"database.include.list": "inventory",
"database.history.kafka.bootstrap.servers": "localhost:9092",
"database.history.kafka.topic": "schema_changes.inventory"
}
}

In the Hashicorp vault, I have stored the secrets in the following path
confluent/secret

But while running source connector with rest call I'm getting the following error
{"error_code":500,"message":"Vault path 'confluent/secrets' was not found"}

We have tired other way also still we are getting error

{"error_code":500,"message":"Vault path 'secret/confluent/secrets' was not found"}
@jcustenborder
Copy link
Owner

That message is coming directly from vault. The library is just bubbling it up. You most likely don't have a secret at that path or have a permissions issue. You need to verify that using the token you've configured you can read that secret.

Make sure that this command works with the token in your config.

On your side make sure that you can run and that returns a secret containing mysqluser and mysqlpass

vault kv get secret/confluent/secrets

For example I have secret/application/example/development in my instance. When I run the following command with the proper token I get the following output.

vault kv get secret/application/example/development

====== Metadata ======
Key              Value
---              -----
created_time     2021-01-30T01:43:13.088708822Z
deletion_time    n/a
destroyed        false
version          3

========== Data ==========
Key                 Value
---                 -----
bootstrap.server    worker-01:30000,worker-02:30000,worker-03:30000
kafka.properties    ssl.endpoint.identification.algorithm=https
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="test" password="test123";
test                asdfasdf

Then I can use this in my config.

${vault:secret/application/example/development:bootstrap.server}

@Kalyan-D
Copy link
Author

Kalyan-D commented Jul 21, 2021 via email

@jcustenborder
Copy link
Owner

None of your images are coming though. Please give the above example a try with vault cli with the token you specified in the config

@Kalyan-D
Copy link
Author

Kalyan-D commented Jul 21, 2021 via email

@Kalyan-D
Copy link
Author

Kalyan-D commented Jul 21, 2021 via email •
edited

@Kalyan-D
Copy link
Author

@jcustenborder

Any update on this issue ?

@Kalyan-D
Copy link
Author

@jcustenborder Could you please look into this issue asap

@Kalyan-D
Copy link
Author

Kalyan-D commented Aug 2, 2021

Hi @jcustenborder
Did you get a chance to look into this?

@WilsonSunBritten
Copy link

I am now encountering this issue myself. I hope to test more tomorrow but to me it looks like a vault engine 1 vs 2 issue. The newest release of https://github.com/BetterCloud/vault-java-driver that this project uses defaults to vault engine 2, while the vault path you provided looks like an engine v1 path. I have to do some more testing but it looks like either a config for a secret path map can be included, or a global engine version flag can be included(This repo needs to update to provide those config options, but BettterCloud does have them).

@WilsonSunBritten WilsonSunBritten linked a pull request Nov 2, 2021 that will close this issue
Open
@AviMualem
Copy link

AviMualem commented May 15, 2022 •
edited

im having the exact same situation. getting vault path not found, for secrets i can get manually by same path.
curl works for me for the same path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add Vault engine specifying configs WilsonSunBritten/kafka-config-provider-vault
4 participants
@jcustenborder @AviMualem @WilsonSunBritten @Kalyan-D