bug: External topics not refreshed on page reload

[andormarkus]

Hi Team,

We are configuring our topic with Terraform or Kafka CLI therefore these topic are "External" topic.

I have deleted topic "navisioncompany" and created new topic "navision.Company".
This new topic show up in Kafka client without error. However these changes are not picked up by Kafka UI. To track these changes I have to restart the docker container.

Kafka CLI Client

Kafka UI

Thanks,
Andor

[germanosin]

Thank you for your issue. Will try to fix it soon.

[germanosin]

@andormarkus Kafka UI uses internal cache for rest requests. By default it updates once in a 30 seconds. I just checked that new topic created outside of Kafka UI successfully appeared in 30 seconds. Could you please verify that this was an issue for you?

[andormarkus]

@germanosin We are running it inside a docker container and it might cause the issue. If we add or delete any topic it does not show up until docker restarted. I have waited 30 minutes and nothing, while on CLI I can see the new/deleted topics. Tried incognito window, delete browser cache etc.

We are running it on AWS t3a.micro (1gb ram, Amazon Linux 2) instance inside docker. This is the only container running on this instance and cloudwatch agent. Based on htop kafka-ui-api.jar consumes 600mb/1024mb memory. Without kafka ui the base memory usage in ~200mb

This is the user data how we are launching the kafka ui.

#!/bin/sh
export PATH=/usr/local/bin:$PATH

##----------------------------------------------------------------------------------------
# Install the necessary packages

yum install jq htop -y

# Install docker
yum install docker -y
service docker start

groupadd docker
usermod -a -G docker ec2-user

# Install docker-compose
curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
chown root:docker /usr/local/bin/docker-compose

#----------------------------------------------------------------------------------------
# Create Kafka UI docker

KAFKA_UI=/home/ec2-user/kafka_ui
mkdir $KAFKA_UI

cd $KAFKA_UI

# write out the docker-compose file
cat <<EOF >docker-compose.yml
version: '2'
services:
  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
     - "80:8080"
    restart: always
    environment:
EOF

#----------------------------------------------------------------------------------------
# Get Kafka information

CLUSTER_ARNS=$(aws kafka list-clusters --region=eu-central-1 | jq ".ClusterInfoList | .[] | .ClusterArn" | tr -d '"')
CLUSTER_ARNS=($CLUSTER_ARNS)

INDEX=0
for ARN in ${CLUSTER_ARNS[@]}; do
    NAME_LONG=$(echo $ARN | grep -o "cluster\/.*\/")
    NAME=${NAME_LONG:8:-1}

    BROKER_URL=$(aws kafka get-bootstrap-brokers --cluster-arn=$ARN --region=eu-central-1 | jq ".BootstrapBrokerString" | tr -d '"')
    ZOOKEEPER_URL=$(aws kafka describe-cluster --cluster-arn=$ARN --region=eu-central-1 | jq ".ClusterInfo | .ZookeeperConnectString" | tr -d '"')

    echo "     - KAFKA_CLUSTERS_${INDEX}_NAME=${NAME}" >>docker-compose.yml
    echo "     - KAFKA_CLUSTERS_${INDEX}_BOOTSTRAPSERVERS=${BROKER_URL}" >>docker-compose.yml
    echo "     - KAFKA_CLUSTERS_${INDEX}_ZOOKEEPER=${ZOOKEEPER_URL}" >>docker-compose.yml

    INDEX=$(expr $INDEX + 1)
done

#Fix docker-compose ownership
chown ec2-user:ec2-user docker-compose.yml

# Start the container
docker-compose up -d

Please let me know if you need more information

[andormarkus]

@germanosin

I have done further testing. The topics automatically shows up after 30s and page reload.

I have also noticed when the kafka-ui-api.jar runs out of system memory (1gb) which takes 5h while no one uses it and just idling. See screenshot in my previous comment. When the system out of memory than topic refresh stop and no new topic shows up and deleted disappears. Message can not be loaded in any topic as well, just buffers. I don't know what causes this out of memory issues.

[andormarkus]

@germanosin

Hello,

I made further testing, this case Kafka UI is not connected to any Kafka cluster. Same t3a.micro (1gb ram, Amazon Linux 2) instance was used. With the bootstrap script you can recreate the same situation.

Please let me know if you need more information.

Bootstrap script on AWS

#!/bin/sh
export PATH=/usr/local/bin:$PATH

# Install docker
yum install docker -y
service docker start

groupadd docker
usermod -a -G docker ec2-user

# Install docker-compose
curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
chown root:docker /usr/local/bin/docker-compose

#----------------------------------------------------------------------------------------
# Create Kafka UI docker

KAFKA_UI=/home/ec2-user/kafka_ui
mkdir $KAFKA_UI

cd $KAFKA_UI

# write out the docker-compose file
cat <<EOF >docker-compose.yml
version: '2'
services:
  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
     - "80:8080"
    restart: always
EOF

#Fix docker-compose ownership
chown ec2-user:ec2-user docker-compose.yml

# Start the container
docker-compose up -d

Here is the memory and disk usage profile of the instance.

[germanosin]

@andormarkus this issue should be fixed in 0.0.10. Could you please check it on your side?

[andormarkus]

Hi @germanosin

Please can you update docker hub tags?
Latest tag belongs to 0.0.9

Thanks

[andormarkus]

Hi All,

To work around the memory issue on 0.0.9 we had a cron job which restarted the container in every 4 hours. This explains sawtooth wave. For 0.0.10 I have disabled the cron job. I will get back once more data collected.

[andormarkus]

Here is 7 days ram utilisation