Not able to run pulsar postgres jdbc sink

[SapnaGirdhani1]

Describe the bug
I am using Docker to create pulsar_postgres_jdbc_sink but it is not running. Status shows running as false.

To Reproduce
I followed this document : https://pulsar.apache.org/docs/en/io-quickstart/#create-a-jdbc-sink to create jdbc sink
I used below command to create a sink :
$ bin/pulsar-admin sinks create
--archive ./connectors/pulsar-io-jdbc-postgres-2.8.1.nar
--inputs my-topic
--name jdbc-postgres-sink
--sink-config-file ./connectors/pulsar-postgres-jdbc-sink.yaml
--parallelism 1

This is my yaml (pulsar-postgres-jdbc-sink.yaml) file content :
tenant: "public"
namespace: "default"
name: "jdbc-postgres-sink"
topicName: "persistent://public/default/my-topic"
sinkType: "jdbc-postgres"
configs:
userName: "postgres"
password: "password"
jdbcUrl: "jdbc:postgresql://localhost:5432/pulsar_postgres_jdbc_sink"
tableName: "pulsar_postgres_jdbc_sink"
After creation of sink , I didn't see any data inside PostgreSQL database and the sink running status is "false" (screenshot attached)

Expected behavior
pulsar_postgres_jdbc_sink should be running

Screenshots

Desktop (please complete the following information):

• OS: Windows 10
• RAM: 8 gb

Additional context
I've discussed this issue on official pulsar slack channel and got to know that others are also facing same challenge, even on Macbook too. It is observed that one can not run postgresql and pulsar and docker and chrome without running out of memory. Please help on the same. Thanks!

[shoothzj]

is there any error in the log ?

[SapnaGirdhani1]

This is what I could find in logs:

10:01:31.534 [function-timer-thread-91-1] ERROR org.apache.pulsar.functions.runtime.process.ProcessRuntime - Health check failed for jdbc-postgres-sink-0
java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) ~[?:?]
at org.apache.pulsar.functions.runtime.process.ProcessRuntime.lambda$start$1(ProcessRuntime.java:186) ~[org.apache.pulsar-pulsar-functions-runtime-2.8.1.jar:2.8.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.66.Final.jar:4.1.66.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]

[tuteng]

@SapnaGirdhani1 I think you can refer to this document for debugging https://pulsar.apache.org/docs/en/io-debug/#debug-in-localrun-mode

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[tisonkun]

Cannot reproduce on master:

$ bin/pulsar-admin sinks create \
    --archive ./connectors/pulsar-io-jdbc-postgres-3.1.0-SNAPSHOT.nar \ 
    --inputs pulsar-postgres-jdbc-sink-topic \
    --name pulsar-postgres-jdbc-sink \
    --sink-config-file ./connectors/pulsar-postgres-jdbc-sink.yaml \ 
    --parallelism 1
Created successfully
$ bin/pulsar-admin sinks list \
    --tenant public \
    --namespace default
[
  "pulsar-postgres-jdbc-sink"
]
$ bin/pulsar-admin sinks get \
  --tenant public \
  --namespace default \
  --name pulsar-postgres-jdbc-sink
{
  "tenant": "public",
  "namespace": "default",
  "name": "pulsar-postgres-jdbc-sink",
  "className": "org.apache.pulsar.io.jdbc.PostgresJdbcAutoSchemaSink",
  "sourceSubscriptionPosition": "Latest",
  "inputs": [
    "pulsar-postgres-jdbc-sink-topic"
  ],
  "inputSpecs": {
    "pulsar-postgres-jdbc-sink-topic": {
      "isRegexPattern": false,
      "schemaProperties": {},
      "consumerProperties": {},
      "poolMessages": false
    }
  },
  "configs": {
    "jdbcUrl": "jdbc:postgresql://localhost:5432/postgres",
    "userName": "chenzili",
    "tableName": "pulsar_postgres_jdbc_sink"
  },
  "parallelism": 1,
  "processingGuarantees": "ATLEAST_ONCE",
  "retainOrdering": false,
  "retainKeyOrdering": false,
  "resources": {
    "cpu": 1.0,
    "ram": 1073741824,
    "disk": 10737418240
  },
  "autoAck": true,
  "cleanupSubscription": true
}
$ bin/pulsar-admin sinks status \
  --tenant public \
  --namespace default \
  --name pulsar-postgres-jdbc-sink
{
  "numInstances" : 1,
  "numRunning" : 1,
  "instances" : [ {
    "instanceId" : 0,
    "status" : {
      "running" : true,
      "error" : "",
      "numRestarts" : 0,
      "numReadFromPulsar" : 0,
      "numSystemExceptions" : 0,
      "latestSystemExceptions" : [ ],
      "numSinkExceptions" : 0,
      "latestSinkExceptions" : [ ],
      "numWrittenToSink" : 0,
      "lastReceivedTime" : 0,
      "workerId" : "c-standalone-fw-localhost-8080"
    }
  } ]
}