ERROR with postgreSQLSink with Cygnus on Docker

[agaldemas]

Cygnus is deployed with Docker, mongo sink, sth sink, mysql sink work OK and persist the data
but postgreSQL sink give an error :

time=2018-02-19T13:59:42.355Z | lvl=INFO | corr=0e6caa90-157d-11e8-b9b3-0242ac11000a | trans=83810acc-a2b9-46c6-8b52-c795e30cb645 | srv=testservice | subsrv=/testalain | comp=cygnus-ngsi | op=persistAggregation | msg=com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink[490] : [postgresql-sink] Persisting data at NGSIPostgreSQLSink. Schema (testservice), Table (testalain_device_001_thing), Fields ((recvTimeTs,recvTime,fiwareServicePath,entityId,entityType,attrName,attrType,attrValue,attrMd)), Values (('1519048749408','2018-02-19T13:59:09.408Z','/testalain','device_001','thing','temperature','Float','27.3','[{"name":"TimeInstant","type":"ISO8601","value":"2018-02-19T13:59:09.141Z"}]'))
time=2018-02-19T13:59:42.363Z | lvl=ERROR | corr=0e6caa90-157d-11e8-b9b3-0242ac11000a | trans=83810acc-a2b9-46c6-8b52-c795e30cb645 | srv=testservice | subsrv=/testalain | comp=cygnus-ngsi | op=processNewBatches | msg=com.telefonica.iot.cygnus.sinks.NGSISink[569] : CygnusPersistenceError. -, null. Stack trace: [com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink.persistAggregation(NGSIPostgreSQLSink.java:504), com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink.persistBatch(NGSIPostgreSQLSink.java:231), com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:558), com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370), org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68), org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147), java.lang.Thread.run(Thread.java:748)]

this error occurs even with cygnus-ngsi.sinks.postgresql.backend.enable_cache = true, like proposed as a solution in (#1446)
#1446
also someone report the same issue a few days ago:
https://stackoverflow.com/questions/48787905/cygnus-ngsi-not-able-to-save-data-in-postgresql-using-docker

my configuration uses postgres:latest image and telefonicaiot/fiware-cygnus:latest
this error has been reported several times, but no solution has been provided

thanks in advance for help or issue fixing

[fgalan]

Yes, this issues could be a particular aspects of the more general #1464

[agaldemas]

OK, but the issue doesn't seem to resume to a Docker build issue.
the PostgreSQLSink initialze correctly at cygnus start, it fails at the step to create the schema and the tables in the postgres database
my agent.conf file is :

cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = mysql-sink mongo-sink sth-sink postgresql-sink
cygnus-ngsi.channels = mysql-channel mongo-channel sth-channel postgresql-channel

cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = mysql-channel mongo-channel sth-channel postgresql-channel
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /opt/apache-flume/conf/grouping_rules.conf

cygnus-ngsi.sinks.mysql-sink.type = com.telefonica.iot.cygnus.sinks.NGSIMySQLSink
cygnus-ngsi.sinks.mysql-sink.channel = mysql-channel
cygnus-ngsi.sinks.mysql-sink.enable_grouping = false
cygnus-ngsi.sinks.mysql-sink.enable_lowercase = false
cygnus-ngsi.sinks.mysql-sink.mysql_host = mysql
cygnus-ngsi.sinks.mysql-sink.mysql_port = 3306
cygnus-ngsi.sinks.mysql-sink.mysql_username = root
cygnus-ngsi.sinks.mysql-sink.mysql_password = mysql
cygnus-ngsi.sinks.mysql-sink.attr_persistence = row
cygnus-ngsi.sinks.mysql-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.mysql-sink.batch_size = 10
cygnus-ngsi.sinks.mysql-sink.batch_timeout = 5
cygnus-ngsi.sinks.mysql-sink.batch_ttl = 0

cygnus-ngsi.sinks.mongo-sink.type = com.telefonica.iot.cygnus.sinks.NGSIMongoSink
cygnus-ngsi.sinks.mongo-sink.channel = mongo-channel
cygnus-ngsi.sinks.mongo-sink.enable_grouping = false
cygnus-ngsi.sinks.mongo-sink.enable_lowercase = false
cygnus-ngsi.sinks.mongo-sink.mongo_hosts = mongo:27017
cygnus-ngsi.sinks.mongo-sink.mongo_username =
cygnus-ngsi.sinks.mongo-sink.mongo_password =
cygnus-ngsi.sinks.mongo-sink.db_prefix = sth_
cygnus-ngsi.sinks.mongo-sink.collection_prefix = sth_
cygnus-ngsi.sinks.mongo-sink.should_hash = false
cygnus-ngsi.sinks.mongo-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.mongo-sink.attr_persistence = row
cygnus-ngsi.sinks.mongo-sink.batch_size = 10
cygnus-ngsi.sinks.mongo-sink.batch_timeout = 5
cygnus-ngsi.sinks.mongo-sink.batch_ttl = 0
cygnus-ngsi.sinks.mongo-sink.data_expiration = 0
cygnus-ngsi.sinks.mongo-sink.collections_size = 0
cygnus-ngsi.sinks.mongo-sink.max_documents = 0

cygnus-ngsi.sinks.sth-sink.type = com.telefonica.iot.cygnus.sinks.NGSISTHSink
cygnus-ngsi.sinks.sth-sink.channel = sth-channel
cygnus-ngsi.sinks.sth-sink.enable_grouping = false
cygnus-ngsi.sinks.sth-sink.enable_lowercase = false
cygnus-ngsi.sinks.sth-sink.mongo_hosts = mongo:27017
cygnus-ngsi.sinks.sth-sink.mongo_username =
cygnus-ngsi.sinks.sth-sink.mongo_password =
cygnus-ngsi.sinks.sth-sink.db_prefix = sth_
cygnus-ngsi.sinks.sth-sink.collection_prefix = sth_
cygnus-ngsi.sinks.sth-sink.should_hash = false
cygnus-ngsi.sinks.sth-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.sth-sink.attr_persistence = row
cygnus-ngsi.sinks.sth-sink.batch_size = 10
cygnus-ngsi.sinks.sth-sink.batch_timeout = 5
cygnus-ngsi.sinks.sth-sink.batch_ttl = 0
cygnus-ngsi.sinks.sth-sink.data_expiration = 0
cygnus-ngsi.sinks.sth-sink.collections_size = 0
cygnus-ngsi.sinks.sth-sink.max_documents = 0

============================================

# NGSIPostgreSQLSink configuration
cygnus-ngsi.sinks.postgresql-sink.type = com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink
cygnus-ngsi.sinks.postgresql-sink.channel = postgresql-channel
cygnus-ngsi.sinks.postgresql-sink.enable_encoding = false
cygnus-ngsi.sinks.postgresql-sink.enable_name_mappings = false
cygnus-ngsi.sinks.postgresql-sink.enable_grouping = false
cygnus-ngsi.sinks.postgresql-sink.enable_lowercase = false
cygnus-ngsi.sinks.postgresql-sink.postgresql_host = postgres
cygnus-ngsi.sinks.postgresql-sink.postgresql_port = 5432
cygnus-ngsi.sinks.postgresql-sink.postgresql_database = test
cygnus-ngsi.sinks.postgresql-sink.postgresql_username = toto
cygnus-ngsi.sinks.postgresql-sink.postgresql_password = postgres
cygnus-ngsi.sinks.postgresql-sink.attr_persistence = row
cygnus-ngsi.sinks.postgresql-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.postgresql-sink.batch_size = 10
cygnus-ngsi.sinks.postgresql-sink.batch_timeout = 10
cygnus-ngsi.sinks.postgresql-sink.batch_ttl = 0
cygnus-ngsi.sinks.postgresql.backend.enable_cache = true

cygnus-ngsi.channels.mysql-channel.type = com.telefonica.iot.cygnus.channels.CygnusMemoryChannel
cygnus-ngsi.channels.mysql-channel.capacity = 1000
cygnus-ngsi.channels.mysql-channel.transactionCapacity = 100

cygnus-ngsi.channels.mongo-channel.type = com.telefonica.iot.cygnus.channels.CygnusMemoryChannel
cygnus-ngsi.channels.mongo-channel.capacity = 1000
cygnus-ngsi.channels.mongo-channel.transactionCapacity = 100

cygnus-ngsi.channels.sth-channel.type = com.telefonica.iot.cygnus.channels.CygnusMemoryChannel
cygnus-ngsi.channels.sth-channel.capacity = 1000
cygnus-ngsi.channels.sth-channel.transactionCapacity = 100

cygnus-ngsi.channels.postgresql-channel.type = com.telefonica.iot.cygnus.channels.CygnusMemoryChannel
cygnus-ngsi.channels.postgresql-channel.capacity = 1000
cygnus-ngsi.channels.postgresql-channel.transactionCapacity = 100

All the sinks work OK except the postgres one
the database is created at postgres container creation, and can be accessed with the provided user/password
the postgres hostname is resolved in the cygnus container
I also created the schema with the name of the Fiware-service, to test, the same error occur...

[AlvaroVega]

Can we close this issue?