Skip to content

Commit

Permalink
trie: stop indexing empty directory nodes
Browse files Browse the repository at this point in the history
There is a strange bug in trie that seems related to indexing empty directory nodes. The actual root cause
is still unknown to me and I fail to reproduce the issue. In our production, we notice the issue associates
with a cron job for removing files (which would causes the generation of empty directories).

Considering that empty directories doesnt produce much value in go-carbon/graphite, excluding it from
index should cause no harm. And this change also appears to resolve the issue on our production.
  • Loading branch information
bom-d-van committed Jan 11, 2022
1 parent 16250e7 commit 67446d3
Showing 1 changed file with 21 additions and 15 deletions.
36 changes: 21 additions & 15 deletions carbonserver/carbonserver.go
Original file line number Diff line number Diff line change
Expand Up @@ -990,21 +990,27 @@ func (listener *CarbonserverListener) updateFileList(dir string, cacheMetricName
delete(cacheMetricNames, trimmedName)
} else {
if listener.trieIndex {
var dataPoints int64
if isFullMetric && listener.estimateSize != nil {
m := strings.ReplaceAll(trimmedName, "/", ".")
m = m[1 : len(m)-4]
_, dataPoints = listener.estimateSize(m)
}

var physicalSize = info.Size()
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
physicalSize = stat.Blocks * 512
}

if err := trieIdx.insert(trimmedName, info.Size(), physicalSize, dataPoints); err != nil {
// It's better to just log an error than stop indexing
listener.logTrieInsertError(logger, "updateFileList.trie: failed to index path", trimmedName, err)
// WHY:
// * this would only affects empty directories
// * indexing empty directories causes an strange bug in trie index
// * empty dir isn't useful (at least most of the time)?
if isFullMetric {
var dataPoints int64
if listener.estimateSize != nil {
m := strings.ReplaceAll(trimmedName, "/", ".")
m = m[1 : len(m)-4]
_, dataPoints = listener.estimateSize(m)
}

var physicalSize = info.Size()
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
physicalSize = stat.Blocks * 512
}

if err := trieIdx.insert(trimmedName, info.Size(), physicalSize, dataPoints); err != nil {
// It's better to just log an error than stop indexing
listener.logTrieInsertError(logger, "updateFileList.trie: failed to index path", trimmedName, err)
}
}
} else {
files = append(files, trimmedName)
Expand Down

0 comments on commit 67446d3

Please sign in to comment.