pulsar dashboard collector encountered error message

[ltamber]

we deployed a cluster on bare metal, and running pulsar dashboard in docker, then we found some exceptions in /var/log/supervisor/collector-error.log, Is there a problem with this exception?

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "stats_namespace_clusters_namespace_id_cluster_id_2d7f67e8_uniq"
DETAIL: Key (namespace_id, cluster_id)=(17, 1) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "./collector.py", line 64, in fetch_broker_stats
_fetch_broker_stats(cluster, broker_url, timestamp)
File "./collector.py", line 104, in _fetch_broker_stats
namespace.clusters.add(cluster)
File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/related_descriptors.py", line 934, in add
self._add_items(self.source_field_name, self.target_field_name, *objs)
File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/related_descriptors.py", line 1103, in _add_items
for obj_id in new_ids
File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 443, in bulk_create
ids = self._batched_insert(objs_without_pk, fields, batch_size)
File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1096, in _batched_insert
inserted_id = self._insert(item, fields=fields, using=self.db, return_id=True)
File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1079, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1112, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 94, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python3.7/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "stats_namespace_clusters_namespace_id_cluster_id_2d7f67e8_uniq"
DETAIL: Key (namespace_id, cluster_id)=(17, 1) already exists.

Traceback (most recent call last):
File "./collector.py", line 64, in fetch_broker_stats
_fetch_broker_stats(cluster, broker_url, timestamp)
File "./collector.py", line 336, in _fetch_broker_stats
timestamp=timestamp)
File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 464, in get_or_create
return self.get(**lookup), False
File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 384, in get
(self.model._meta.object_name, num)
stats.models.MultipleObjectsReturned: get() returned more than one Namespace -- it returned 2!
Traceback (most recent call last):
File "./collector.py", line 64, in fetch_broker_stats
_fetch_broker_stats(cluster, broker_url, timestamp)
File "./collector.py", line 336, in _fetch_broker_stats
timestamp=timestamp)
File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 464, in get_or_create
return self.get(**lookup), False
File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 384, in get
(self.model._meta.object_name, num)
stats.models.MultipleObjectsReturned: get() returned more than one Namespace -- it returned 2!
Process Process-11:
multiprocessing.pool.RemoteTraceback:

environment:

• OS: centos7.0
• Docker version: 19.03.2

[jiazhai]

@tuteng would you please help take a look of this issue?

[tuteng]

It seems that there are more namespaces. You can check whether there are many identical namespaces in your cluster. In theory, in one environment, multiple cluster namespaces should be unique in one environment.

[ltamber]

@tuteng I have used the pulsar-admin tool to check namespace, it seem thats there is only one namespace in my cluster.
$ ./pulsar-admin tenants list
tenantTest
$ ./pulsar-admin namespaces list tenantTest
tenantTest/namespaceTest

[tuteng]

I'm very sorry for the late reply. I didn't have your problem tested locally. Can you show me some steps of your deployment or read some more detailed log information?

[jiazhai]

@ltamber Would you please help provide detailed steps for reproduce? so @tuteng could help on it.

[ltamber]

@jiazhai sorry, it has been a long time since we last used pulsar dashboard, and I don't remember well about the steps of the deployment. we are currently using pulsar-manager.

[tuteng]

@ltamber Thank you for your feedback. Let's close this issue first. If you have any questions, you can consider reopen it.