bug: Unable to add CA certificate for ELK plugin destination.

[Mastix95]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Using CloudQuery sync with Azure as source plugin and Elasticsearch as destination plugin with the following yaml configuration files:

Azure:

kind: source
spec:
  name: "azure"
  registry: "github"
  path: "cloudquery/azure"
  version: "v4.3.0"
  destinations: ["elasticsearch"]
  spec:
    subscriptions: ["00000000-0000-0000-0000-00000000"]_

Elastisearch:

kind: destination
spec:
  name: elasticsearch
  path: cloudquery/elasticsearch
  version: "v1.1.5"
  write_mode: "overwrite-delete-stale"
  spec:
    addresses: ["https://elasticip:port"] # optional
    api_key: "api-key-goes-here"
    ca_cert: '${file:/path/to/certificate/file/certificate.pem}'
    certificate_fingerprint: "00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00"_

With this configuration files, the sync flag is giving me the following error:

_Error: failed to sync v1 source azure: failed to initialize destination plugin client for elasticsearch: destination configure: failed to call Configure: rpc error: code = Unknown desc = failed to create Elasticsearch client: error creating transport: unable to add CA certificate_

Tried with double quotes on the file variable substitution, as well as just using the path as a value for ca_certs

Expected Behavior

The sync works properly, sending the source plugin data to the destination.

CloudQuery (redacted) config

Azure:

kind: source
spec:
  name: "azure"
  registry: "github"
  path: "cloudquery/azure"
  version: "v4.3.0"
  destinations: ["elasticsearch"]
  spec:
    subscriptions: ["00000000-0000-0000-0000-00000000"]_

Elastisearch:

kind: destination
spec:
  name: elasticsearch
  path: cloudquery/elasticsearch
  version: "v1.1.5"
  write_mode: "overwrite-delete-stale"
  spec:
    addresses: ["https://elasticip:port"] # optional
    api_key: "api-key-goes-here"
    ca_cert: '${file:/path/to/certificate/file/certificate.pem}'
    certificate_fingerprint: "00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00"_

Steps To Reproduce

1. In a linux (Ubuntu 20.04) environment
2. With the config pasted in the previous step.
3. Execute "./cloudquery sync azurelab.yaml elasticsearch.yaml"
4. See error:

_Error: failed to sync v1 source azure: failed to initialize destination plugin client for elasticsearch: destination configure: failed to call Configure: rpc error: code = Unknown desc = failed to create Elasticsearch client: error creating transport: unable to add CA certificate_

CloudQuery (redacted) logs

2023-06-01T16:01:09Z INF Loading spec(s) args=["azurelab.yaml","elasticsearch.yaml"] module=cli
2023-06-01T16:01:10Z INF Source plugin server listening address=/tmp/cq-LdtxJGjOpLxlFodM.sock module=cli
2023-06-01T16:01:11Z INF Start sync destinations=["elasticsearch (v1.1.5)"] module=cli source="azure (v4.3.0)" sync_time=2023-06-01T16:01:11Z
2023-06-01T16:01:11Z INF Destination plugin server listening address=/tmp/cq-zikkUzZEDSuHZASc.sock module=cli
2023-06-01T16:01:12Z INF started call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-01T16:01:12Z grpc.time_ms=0.025 module=cli peer.address=@ protocol=grpc
2023-06-01T16:01:12Z ERR finished call grpc.code=Unknown grpc.component=server grpc.error="failed to create Elasticsearch client: error creating transport: unable to add CA certificate" grpc.method=Configure grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-01T16:01:12Z grpc.time_ms=0.372 module=cli peer.address=@ protocol=grpc
2023-06-01T16:01:12Z INF waiting for destination plugin to terminate module=cli
2023-06-01T16:01:12Z INF Got stop signal. Destination plugin server shutting down address=/tmp/cq-zikkUzZEDSuHZASc.sock module=cli signal=interrupt
2023-06-01T16:01:12Z INF End sync destinations=["elasticsearch (v1.1.5)"] module=cli source="azure (v4.3.0)" sync_time=2023-06-01T16:01:11Z
2023-06-01T16:01:12Z INF waiting for source plugin to terminate module=cli
2023-06-01T16:01:12Z INF Got stop signal. Source plugin server shutting down address=/tmp/cq-LdtxJGjOpLxlFodM.sock module=cli signal=interrupt
2023-06-01T16:01:12Z INF Sending sync summary to analyticsv1.cloudquery.io:443 module=cli
2023-06-01T16:01:12Z ERR exiting with error error="failed to sync v1 source azure: failed to initialize destination plugin client for elasticsearch: destination configure: failed to call Configure: rpc error: code = Unknown desc = failed to create Elasticsearch client: error creating transport: unable to add CA certificate" module=cli

CloudQuery version

2.5.3

Additional Context

No response

Pull request (optional)

• I can submit a pull request

[github-actions]

Thanks for reporting this issue 👍
You can reach us via Discord too.
If you enjoy using this project, please consider starring it for support

[disq]

Hopefully fixed, we'll release a new cli version soon for you to try.

[disq]

@Mastix95 Released with CloudQuery v3.5.1 can you give that a try? You should also get rid of the single quotes around the var so it should look like:

ca_cert: ${file:/path/to/certificate/file/certificate.pem}

We also updated the Variable Substitution topic in the docs to that effect.

[Mastix95]

Hi @disq, everything is working fine now with the new cli version!

Thanks for the support.