data-connector-agent errors

[AntouanK]

Hello there.

Having an issue understanding why the "data-connector-agent" is only logging errors.

I used this docker-compose :

version: "3.6"

services:

  postgres:
    image: postgres:15
    restart: always
    volumes:
      - db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: ***

  graphql-engine:
    image: hasura/graphql-engine:v2.28.0
    restart: always
    environment:
      ## postgres database to store Hasura metadata
      HASURA_GRAPHQL_METADATA_DATABASE_URL: postgres://postgres:***@postgres:5432/postgres
      ## this env var can be used to add the above postgres database to Hasura as a data source. this can be removed/updated based on your needs
      PG_DATABASE_URL: postgres://postgres:***@postgres:5432/postgres
      ## enable the console served by server
      HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console
      ## enable debugging mode. It is recommended to disable this in production
      HASURA_GRAPHQL_DEV_MODE: "true"
      HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
      ## uncomment next line to run console offline (i.e load console assets from server instead of CDN)
      # HASURA_GRAPHQL_CONSOLE_ASSETS_DIR: /srv/console-assets
      ## uncomment next line to set an admin secret
      HASURA_GRAPHQL_ADMIN_SECRET: ***
      HASURA_GRAPHQL_METADATA_DEFAULTS: '{"backend_configs":{"dataconnector":{"athena":{"uri":"http://data-connector-agent:8081/api/v1/athena"},"mariadb":{"uri":"http://data-connector-agent:8081/api/v1/mariadb"},"mysql8":{"uri":"http://data-connector-agent:8081/api/v1/mysql"},"oracle":{"uri":"http://data-connector-agent:8081/api/v1/oracle"},"snowflake":{"uri":"http://data-connector-agent:8081/api/v1/snowflake"}}}}'
    depends_on:
      data-connector-agent:
        condition: service_healthy

  data-connector-agent:
    image: hasura/graphql-data-connector:v2.28.0
    restart: always
    environment:
      QUARKUS_LOG_LEVEL: ERROR # FATAL, ERROR, WARN, INFO, DEBUG, TRACE
      ## https://quarkus.io/guides/opentelemetry#configuration-reference
      QUARKUS_OPENTELEMETRY_ENABLED: "false"
      ## QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_ENDPOINT: http://jaeger:4317
    healthcheck:
      test:
        [
          "CMD",
          "curl",
          "-f",
          "http://localhost:8081/api/v1/athena/health"
        ]
      interval: 5s
      timeout: 10s
      retries: 5
      start_period: 5s

volumes:
  db_data:


networks:
  default:
    external: true
    name: proxy-manager

Started from fresh and this is the logs I get :

...
hasura-graphql-engine-1        | {"detail":{"info":"Help us improve Hasura! The graphql-engine server collects anonymized usage stats which allows us to keep improving Hasura at warp speed. To read more or opt-out, visit https://hasura.io/docs/latest/graphql/core/guides/telemetry.html","kind":"telemetry"},"level":"info","timestamp":"2023-06-27T07:30:50.707+0000","type":"startup"}
hasura-graphql-engine-1        | {"detail":{"info":{"message":"starting API server","time_taken":1.492192123},"kind":"server"},"level":"info","timestamp":"2023-06-27T07:30:50.707+0000","type":"startup"}
hasura-data-connector-agent-1  | {"timestamp":"2023-06-27T07:30:52.225Z","sequence":106,"loggerClassName":"org.jboss.logmanager.Logger","loggerName":"io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter","level":"SEVERE","message":"Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4317","threadName":"OkHttp http://localhost:4317/...","threadId":26,"mdc":{},"ndc":"","hostName":"3a353abec4b8","processName":"super-connector.jar","processId":1}
hasura-data-connector-agent-1  | {"timestamp":"2023-06-27T07:30:57.232Z","sequence":107,"loggerClassName":"org.jboss.logmanager.Logger","loggerName":"io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter","level":"SEVERE","message":"Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4317","threadName":"OkHttp http://localhost:4317/...","threadId":26,"mdc":{},"ndc":"","hostName":"3a353abec4b8","processName":"super-connector.jar","processId":1}
hasura-data-connector-agent-1  | {"timestamp":"2023-06-27T07:31:02.235Z","sequence":108,"loggerClassName":"org.jboss.logmanager.Logger","loggerName":"io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter","level":"SEVERE","message":"Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4317","threadName":"OkHttp http://localhost:4317/...","threadId":26,"mdc":{},"ndc":"","hostName":"3a353abec4b8","processName":"super-connector.jar","processId":1}

What might be the issue?
What is supposed to be at 127.0.0.1:4317?

[adas98012]

@AntouanK
I see that your "open telemetry tracer" is commented out, e.g.
## QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_ENDPOINT: http://jaeger:4317
So, my guess is that the connector agent is trying to connect to the same on localhost (on port 4317).

[manasag]

Probably need to change QUARKUS_LOG_LEVEL: ERROR to QUARKUS_LOG_LEVEL: INFO to print more types of logs.
The OpenTelemetry exporter seems to be disabled as per docker compose, so it should log about failed to export spans. We will check into this and update. cc @codedmart

[li-dennis]

Using the default docker-compose with the docs generates this error as well https://github.com/hasura/graphql-engine/blob/master/install-manifests/docker-compose/docker-compose.yaml

data-connector-agent_1  | {"timestamp":"2023-09-07T17:02:05.366Z","sequence":126,"loggerClassName":"org.jboss.logmanager.Logger","loggerName":"io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter","level":"SEVERE","message":"Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4317","threadName":"OkHttp http://localhost:4317/...","threadId":30,"mdc":{},"ndc":"","hostName":"c10230621d38","processName":"super-connector.jar","processId":1}

[Nexyll]

I'm facing the same problem, following this deployment guide : https://hasura.io/docs/latest/deployment/deployment-guides/docker/

[Californian]

Has anyone resolved this?