Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug fixes and small refactoring for carbonserver and trie index #416

Merged
merged 4 commits into from Jul 16, 2021

Conversation

bom-d-van
Copy link
Member

It's mainly a misc changes and bug fixes.

Mostly backward-compatible changes except the default values of max-metrics-globbed and max-metrics-rendered.

  1. cache-scan: only init Cache.Shard.adds when the feature is enabled

cache-scan is a feature in go-cabron to enable quicker visibility of new metrics.
However, if the feature is not enabled, no need to cache the new metrics info
in Shard.adds map.

  1. trie: numerious optimization and bug fixes for concurrent and realtime index
  • Fix incorrect index total_runtime calculation
  • Stop caching new metrics between file list scanning (which mean file list scanning interval should be slower than new metrics flushing to disk)
  • Start new file list scanning on startup right after loading files from cache: to avoid serving old file list cache too long
  • Move down the trie_count_nodes_time codes down to be within the indexing "block"
  • Add a new counter cache.droppedRealtimeIndex (use camelCase to stay consistent with the rest of cache.* metrics)
  • Re-initiating trieIndex when failed to load metrics from cache file
  • Some other minor changes here and there
  1. carbonserver: change max-metrics-globbed and max-metrics-rendered defaults (for issue: [FR] Returns error or generating error log when render and glob throttling kicks in #414)

The default values might be too low for some of the users and it is a default introduced mainly
for trie index. This commit change the default to a much bigger value to avoid silently truncated
query result.

  1. carbonserver: change error/debug log destination to access.log for CarbonserverListener.prepareDataProto

cache-scan is a feature in go-cabron to enable quicker visibility of new metrics.
However, if the feature is not enabled, no need to cache the new metrics info
in Shard.adds map.

This change does not alter any existing behaviour in go-carbon.
…e index

* Fix incorrect index total_runtime calculation
* Stop caching new metrics between file list scanning (which mean file list scanning interval should be slower than new metrics flushing to disk)
* Start new file list scanning on startup right after loading files from cache: to avoid serving old file list cache too long
* Move down the trie_count_nodes_time codes down to be within the indexing "block"
* Add a new counter cache.droppedRealtimeIndex (use camelCase to stay consistent with the rest of cache.* metrics)
* Re-initiating trieIndex when failed to load metrics from cache file
* Some other minor changes here and there

Note: mostly backward-compatible changes, except: __file list scanning interval should be slower than new metrics flushing to disk__.
…aults

The default values might be too low for some of the users and it is a default introduced mainly
for trie index. This commit change the default to a much bigger value to avoid silently truncated
query result.

At the same time, logging truncated errors to access.log to improve visibility.
README.md Show resolved Hide resolved
go-carbon.conf.example Show resolved Hide resolved
@bom-d-van bom-d-van merged commit 63cb4c2 into master Jul 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants