Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tuned down log from table manager when running boltdb #1786

Closed
cyriltovena opened this issue Mar 9, 2020 · 6 comments · Fixed by #2030 or #6076
Closed

Tuned down log from table manager when running boltdb #1786

cyriltovena opened this issue Mar 9, 2020 · 6 comments · Fixed by #2030 or #6076
Labels
help wanted We would love help on these issues. Please come help us! keepalive An issue or PR that will be kept alive and never marked as stale. lifecycle/upstream A change must be done to an upstream project first type/enhancement Something existing could be improved

Comments

@cyriltovena
Copy link
Contributor

It seems that the table manager is chatty when running Loki using boltdb.

see transcript.

I am seeing the same issue with a small local instance of loki, which I've recently upgraded to 1.3.0. Every 2 minutes it spams the logs: in fact loki itself is generating more logs than the rest of the source devices!

It appears to be creating tables with the same index_XXXX each time, although not always all of them:

root@loki:/var/log# grep loki-linux-amd64 syslog | grep 19:42: | grep "creating table" | wc -l
61
root@loki:/var/log# grep loki-linux-amd64 syslog | grep 19:44: | grep "creating table" | wc -l
68

Setting -log.level=warn is the only workaround I currently have. I think that pushing these messages down to "debug" level might be better.

Cheers,

Brian.

P.S. Example of one spew:

Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.807745423Z caller=table_manager.go:220 msg="synching tables" expected_tables=100
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808491827Z caller=table_manager.go:363 msg="creating table" table=index_2530
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808563474Z caller=table_manager.go:363 msg="creating table" table=index_2549
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808594246Z caller=table_manager.go:363 msg="creating table" table=index_2554
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808624756Z caller=table_manager.go:363 msg="creating table" table=index_2584
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808654239Z caller=table_manager.go:363 msg="creating table" table=index_2526
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808685498Z caller=table_manager.go:363 msg="creating table" table=index_2548
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808714973Z caller=table_manager.go:363 msg="creating table" table=index_2567
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808745656Z caller=table_manager.go:363 msg="creating table" table=index_2581
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808775198Z caller=table_manager.go:363 msg="creating table" table=index_2573
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808806173Z caller=table_manager.go:363 msg="creating table" table=index_2580
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808924627Z caller=table_manager.go:363 msg="creating table" table=index_2557
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808953787Z caller=table_manager.go:363 msg="creating table" table=index_2562
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.808984762Z caller=table_manager.go:363 msg="creating table" table=index_2558
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809057159Z caller=table_manager.go:363 msg="creating table" table=index_2532
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809088217Z caller=table_manager.go:363 msg="creating table" table=index_2536
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809117834Z caller=table_manager.go:363 msg="creating table" table=index_2538
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809146986Z caller=table_manager.go:363 msg="creating table" table=index_2553
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809176184Z caller=table_manager.go:363 msg="creating table" table=index_2561
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809205524Z caller=table_manager.go:363 msg="creating table" table=index_2563
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809234811Z caller=table_manager.go:363 msg="creating table" table=index_2572
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809264001Z caller=table_manager.go:363 msg="creating table" table=index_2520
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809301913Z caller=table_manager.go:363 msg="creating table" table=index_2521
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809331973Z caller=table_manager.go:363 msg="creating table" table=index_2522
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.80936123Z caller=table_manager.go:363 msg="creating table" table=index_2540
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.80939066Z caller=table_manager.go:363 msg="creating table" table=index_2550
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809420278Z caller=table_manager.go:363 msg="creating table" table=index_2571
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809449243Z caller=table_manager.go:363 msg="creating table" table=index_2579
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809478695Z caller=table_manager.go:363 msg="creating table" table=index_2583
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809507937Z caller=table_manager.go:363 msg="creating table" table=index_2537
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.80953763Z caller=table_manager.go:363 msg="creating table" table=index_2551
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809568364Z caller=table_manager.go:363 msg="creating table" table=index_2544
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809600217Z caller=table_manager.go:363 msg="creating table" table=index_2559
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809629827Z caller=table_manager.go:363 msg="creating table" table=index_2565
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809659579Z caller=table_manager.go:363 msg="creating table" table=index_2587
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809689466Z caller=table_manager.go:363 msg="creating table" table=index_2531
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809718776Z caller=table_manager.go:363 msg="creating table" table=index_2546
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809748334Z caller=table_manager.go:363 msg="creating table" table=index_2552
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809778176Z caller=table_manager.go:363 msg="creating table" table=index_2555
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809807793Z caller=table_manager.go:363 msg="creating table" table=index_2523
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809837118Z caller=table_manager.go:363 msg="creating table" table=index_2524
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809866218Z caller=table_manager.go:363 msg="creating table" table=index_2545
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809896061Z caller=table_manager.go:363 msg="creating table" table=index_2568
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809925483Z caller=table_manager.go:363 msg="creating table" table=index_2586
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.80995651Z caller=table_manager.go:363 msg="creating table" table=index_2535
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.809985865Z caller=table_manager.go:363 msg="creating table" table=index_2564
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.81001585Z caller=table_manager.go:363 msg="creating table" table=index_2578
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810046585Z caller=table_manager.go:363 msg="creating table" table=index_2585
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810076817Z caller=table_manager.go:363 msg="creating table" table=index_2576
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810110035Z caller=table_manager.go:363 msg="creating table" table=index_2582
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810141414Z caller=table_manager.go:363 msg="creating table" table=index_2519
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810171384Z caller=table_manager.go:363 msg="creating table" table=index_2541
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810201744Z caller=table_manager.go:363 msg="creating table" table=index_2574
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810231609Z caller=table_manager.go:363 msg="creating table" table=index_2575
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810261376Z caller=table_manager.go:363 msg="creating table" table=index_2566
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810290596Z caller=table_manager.go:363 msg="creating table" table=index_2570
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810320626Z caller=table_manager.go:363 msg="creating table" table=index_2569
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810370141Z caller=table_manager.go:363 msg="creating table" table=index_2577
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810400043Z caller=table_manager.go:363 msg="creating table" table=index_2528
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810431371Z caller=table_manager.go:363 msg="creating table" table=index_2542
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810461228Z caller=table_manager.go:363 msg="creating table" table=index_2543
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810490485Z caller=table_manager.go:363 msg="creating table" table=index_2560
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.81051981Z caller=table_manager.go:363 msg="creating table" table=index_2547
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810550845Z caller=table_manager.go:363 msg="creating table" table=index_2588
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810589672Z caller=table_manager.go:363 msg="creating table" table=index_2589
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.81061995Z caller=table_manager.go:363 msg="creating table" table=index_2525
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810649597Z caller=table_manager.go:363 msg="creating table" table=index_2527
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810679192Z caller=table_manager.go:363 msg="creating table" table=index_2533
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810708569Z caller=table_manager.go:363 msg="creating table" table=index_2539
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810761362Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2595 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810816846Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2609 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810865049Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2593 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810913528Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2612 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.810963658Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2596 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811011831Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2592 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811059148Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2600 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811108475Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2605 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811156505Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2599 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811206177Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2603 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811270797Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2618 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811321002Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2591 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811405309Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2597 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811458296Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2617 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811507331Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2598 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811556306Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2606 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811604785Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2608 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811652635Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2610 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.81170017Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2607 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811748072Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2602 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811795555Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2604 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811844117Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2611 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811892462Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2614 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811940401Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2594 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.811989219Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2601 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.812036483Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2615 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.812110748Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2590 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.81215901Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2613 read=0 write=0
Mar 6 19:44:17 loki loki-linux-amd64[14433]: level=info ts=2020-03-06T19:44:17.812206358Z caller=table_manager.go:409 msg="provisioned throughput on table, skipping" table=index_2616 read=0 write=0
@cyriltovena cyriltovena added help wanted We would love help on these issues. Please come help us! type/enhancement Something existing could be improved keepalive An issue or PR that will be kept alive and never marked as stale. lifecycle/upstream A change must be done to an upstream project first labels Mar 9, 2020
@cyriltovena
Copy link
Contributor Author

This might require some work on cortex side too.

@adityacs adityacs mentioned this issue Apr 10, 2020
Merged
1 task
@adityacs
Copy link
Contributor

upstream PR is merged. Once this is merged cortexproject/cortex#2551 will update loki to cortex master

@adityacs adityacs mentioned this issue May 1, 2020
Merged
@cyriltovena
Copy link
Contributor Author

Thank you !

cyriltovena pushed a commit to cyriltovena/loki that referenced this issue Jun 11, 2021
@bboreham
Refactor: change index key markers from slice to single byte (grafana…
14f8736 
…#1786)

This is more idiomatic Go and more efficient because it doesn't have
to loop over the slice.

Signed-off-by: Bryan Boreham <bryan@weave.works>
@candlerb
Copy link
Contributor

FYI: as of Loki 2.3.0, and using the default loki-local-config.yaml (except pointing to /data instead of /tmp), I am still seeing excessive log messages from loki every 2 minutes:

...
Sep 21 13:28:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:28:05.116914663Z caller=table_manager.go:476 msg="creating table" table=index_18876
Sep 21 13:28:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:28:05.116926091Z caller=table_manager.go:476 msg="creating table" table=index_18600
Sep 21 13:28:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:28:05.116937095Z caller=table_manager.go:476 msg="creating table" table=index_18807
Sep 21 13:28:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:28:05.11694822Z caller=table_manager.go:476 msg="creating table" table=index_18572
...
Sep 21 13:28:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:28:05.120578533Z caller=table_manager.go:476 msg="creating table" table=index_18862
Sep 21 13:29:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:29:05.099556245Z caller=table_manager.go:171 msg="uploading tables"
Sep 21 13:30:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:30:05.099583345Z caller=table_manager.go:171 msg="uploading tables"
Sep 21 13:30:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:30:05.110699293Z caller=table_manager.go:324 msg="synching tables" expected_tables=333
Sep 21 13:30:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:30:05.11121368Z caller=table_manager.go:476 msg="creating table" table=index_18743
Sep 21 13:30:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:30:05.111259903Z caller=table_manager.go:476 msg="creating table" table=index_18757
Sep 21 13:30:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:30:05.111300781Z caller=table_manager.go:476 msg="creating table" table=index_18765
Sep 21 13:30:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:30:05.111312209Z caller=table_manager.go:476 msg="creating table" table=index_18770
Sep 21 13:30:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:30:05.111323632Z caller=table_manager.go:476 msg="creating table" table=index_18561
Sep 21 13:30:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:30:05.111345327Z caller=table_manager.go:476 msg="creating table" table=index_18656
Sep 21 13:30:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:30:05.111356604Z caller=table_manager.go:476 msg="creating table" table=index_18699
...
Sep 21 13:30:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:30:05.115516004Z caller=table_manager.go:476 msg="creating table" table=index_18687
Sep 21 13:30:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:30:05.115527899Z caller=table_manager.go:476 msg="creating table" table=index_18720
Sep 21 13:31:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:31:05.099535066Z caller=table_manager.go:171 msg="uploading tables"
Sep 21 13:32:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:32:05.099566855Z caller=table_manager.go:171 msg="uploading tables"
Sep 21 13:32:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:32:05.101068587Z caller=table_manager.go:208 msg="syncing tables"
Sep 21 13:32:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:32:05.109572849Z caller=checkpoint.go:612 msg="starting checkpoint"
Sep 21 13:32:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:32:05.109862301Z caller=checkpoint.go:337 msg="attempting checkpoint for" dir=/data/loki/wal/checkpoint.>
Sep 21 13:32:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:32:05.11312512Z caller=table_manager.go:324 msg="synching tables" expected_tables=333
Sep 21 13:32:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:32:05.113565281Z caller=table_manager.go:476 msg="creating table" table=index_18673
Sep 21 13:32:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:32:05.113594746Z caller=table_manager.go:476 msg="creating table" table=index_18831
Sep 21 13:32:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:32:05.113607561Z caller=table_manager.go:476 msg="creating table" table=index_18833
Sep 21 13:32:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:32:05.113618949Z caller=table_manager.go:476 msg="creating table" table=index_18861
Sep 21 13:32:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:32:05.11364656Z caller=table_manager.go:476 msg="creating table" table=index_18646
Sep 21 13:32:05 prometheus loki-linux-amd64[114771]: level=info ts=2021-09-21T12:32:05.113658249Z caller=table_manager.go:476 msg="creating table" table=index_18659
... etc

The only way I can see to suppress these is to run with -log.level=warn

@michaelkebe
Copy link

Please reopen.

The log is hammered with lines like:

level=info ts=2021-10-28T13:00:37.039796668Z caller=table_manager.go:476 msg="creating table" table=index_18872

patman-cp added a commit to patman-cp/loki that referenced this issue May 2, 2022
@patman-cp
table_manager: Change some level.Info() logging to level.Debug()
4fda156 
There is excessive logging of "creating table" and "synching tables" messages, change them
to use level.Debug() instead of level.Info() to limit the output.

Fixes grafana#1786
@patman-cp patman-cp mentioned this issue May 2, 2022
Merged
sandeepsukhani pushed a commit that referenced this issue May 4, 2022
@patman-cp
table_manager: Change some level.Info() logging to level.Debug() (#6076)
b3f6f6b 
There is excessive logging of "creating table" and "synching tables" messages, change them
to use level.Debug() instead of level.Info() to limit the output.

Fixes #1786
@NickLarsenNZ
Copy link

NickLarsenNZ commented Oct 26, 2022 •
edited
Loading

I think this needs reopening. I'm constantly seeing these logs every second. I'm unsure if this is normal behaviour or if it is stuck in a loop. Either way, it seems a little excessive for info level logging.

level=info ts=2022-10-26T17:50:12.126964173Z caller=table_manager.go:534 msg="provisioned throughput on table, skipping" table=loki_chunks_19235 read=2 write=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted We would love help on these issues. Please come help us! keepalive An issue or PR that will be kept alive and never marked as stale. lifecycle/upstream A change must be done to an upstream project first type/enhancement Something existing could be improved
Projects
None yet
Milestone
No milestone
5 participants
@candlerb @michaelkebe @cyriltovena @adityacs @NickLarsenNZ