Log searchs loading times

[dkgs2000]

Check for existing issues

• Completed

How is Dozzle deployed?

Standalone Deployment

Dozzle version

v10.5.3

✅ Command used to run Dozzle

dozzle:
    image: amir20/dozzle:latest
    container_name: dozzle
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - manager_network
    restart: unless-stopped  
    environment:
      - DOZZLE_BASE=/dozzle 

Describe the performance issue you are seeing

tl;dr - when using the regex search in the log screen the loads are long and there are not indication about progression.

Sometimes i search for specific IP in my nginx container logs to follow a single session requests. each search take about 10-15 seconds to populate a proper amount of logs. Sometime the search returns partially and i have no idea if it is done or not, but after a few seconds it fills.

i would like:

1. Have an initial search progress indication
2. Improve the search logic to return partiall results faster
3. have a full log search indication (how much of the original log was searched through by now

Dozzle is great. Thanks!

How many containers, hosts, agents, etc. are you using?

~6 docker containers on 1 host.
Docker v29.4.1
Logs are several MB of text in total

Screenshots, videos, or logs

No response

[amir20]

What is your suggested fix for this?

Have an initial search progress indication

That's already implemented. There is a loader/spinner while searching.

Improve the search logic to return partiall results faster

Search results are streamed and loaded as they come in from API

have a full log search indication (how much of the original log was searched through by now

This one would be a new feature. I am not a UI expert so suggestions welcomed.

To help, I would need actual logs, with search regex patterns so I can test locally.

[amir20]

I did some thinking. I think this could be purely a UI improvement to enhance the experience. But before making improvements, it would be good if you can provide sample logs so I can validate.

[dkgs2000]

What is your suggested fix for this?

Have an initial search progress indication

That's already implemented. There is a loader/spinner while searching.

The majority of the time, the spinner doesn't show (I really don't know why). The rest of the time the spinner doesn't update its percentage during the search. I can provide screenshots of the search I'm using. There is no spinner. just placeholders for log line and colorful line at the end.

Improve the search logic to return partial results faster

Search results are streamed and loaded as they come in from API

obviously, yes. But is it possible to improve the API response rate? I didn't go over the code I'm just saying it bothers me that it takes so much time.

have a full log search indication (how much of the original log was searched through by now

This one would be a new feature. I am not a UI expert so suggestions welcomed.

A simple percent number somewhere that indicates how much of the file was processed would be awesome.

To help, I would need actual logs, with search regex patterns so I can test locally.

I cannot share the actual logs from my production as its a violation of my privacy policy, but I'm sure ANY logs file would reproduce this performance issues under the right condition (some latency, limited memory, etc...).

[amir20]

I made some improvements #4775 but creating a good example wasn't easy. So I am just going to leave it as that.

[amir20]

I am just using Docker API. So I can't improve the speed. The best I can do is provide better UI feedback.

[amir20]

I cannot share the actual logs from my production as its a violation of my privacy policy, but I'm sure ANY logs file would reproduce this performance issues under the right condition (some latency, limited memory, etc...).

I don't need your production logs, anything that reproduces it works. I can't trigger the slowness here because in my logs the matches are common, so search returns instantly. It only drags when the pattern matches a handful of lines in a large log, which is specific to your data.

You said any log file reproduces this under the right conditions, so the fastest path is you giving me one that does. Anonymize a sample or generate one, swap the real ip for something like 203.0.113.5 (a documentation range, leaks nothing), and include the exact search you run. A file that takes 10-15s to search on your end is all I need.

Without a reproducible example I'm just guessing at what's slow, and I'd rather fix the real thing than ship a change that does nothing for you.

[amir20]

Moved this to discussion. If other have this problem then I can take a deeper look.