Fix log spam introduced by branch 'gh62-badrecord-mstate' #66

Merged
merged 1 commit into from Nov 14, 2012

Conversation

Projects
None yet
2 participants
Contributor

slfritchie commented Oct 27, 2012

So, I worried a bit about the possibility of spamminess of
the changes made by branch 'gh62-badrecord-mstate' at the time
I wrote it, but not enough. In a pull-the-power-cord failure,
the next Riak start included tens of thousands of error messages.
See example below.

Fix: Introduce a cutoff of 20 CRC errors when folding over any
single file.

( echo "line-count path bytes-skipped" ; grep 'skip' ALL.console.log | awk '{print $11, $15}' | uniq -c | sort -k 2 | sed 's/^  *//' ) | sed 's/ /, /g'
42918, /var/lib/riak/bitcask/1019054852592055355075616182186939527377281089536/3.bitcask.data, 14
1, /var/lib/riak/bitcask/1183188337253240741397319068673631468061296951296/3.bitcask.data, 1049176
34757, /var/lib/riak/bitcask/1218869529570889738423776217909868846470865616896/3.bitcask.data, 14
23, /var/lib/riak/bitcask/1247414483425008936044941937298858749198520549376/3.bitcask.data, 14
1, /var/lib/riak/bitcask/1283095675742657933071399086535096127608089214976/3.bitcask.data, 1049177
23444, /var/lib/riak/bitcask/1283095675742657933071399086535096127608089214976/3.bitcask.data, 14
1, /var/lib/riak/bitcask/1290231914206187732476690516382343603290002948096/3.bitcask.data, 1049177
36198, /var/lib/riak/bitcask/1290231914206187732476690516382343603290002948096/3.bitcask.data, 14
292, /var/lib/riak/bitcask/248341098530837019304141758684212153730597912576/3.bitcask.data, 14
258, /var/lib/riak/bitcask/398202106264962806815261785476409143050786308096/3.bitcask.data, 14
165, /var/lib/riak/bitcask/419610821655552205031136075018151570096527507456/3.bitcask.data, 14
156, /var/lib/riak/bitcask/433883298582611803841718934712646521460354973696/3.bitcask.data, 14
1, /var/lib/riak/bitcask/533790637072028995515798952574111181007147237376/3.bitcask.data, 1049176
3954, /var/lib/riak/bitcask/533790637072028995515798952574111181007147237376/3.bitcask.data, 14
1, /var/lib/riak/bitcask/626561737097916387784587540588328364872025767936/3.bitcask.data, 1049174
77531, /var/lib/riak/bitcask/62798898479062234766564582655777786000840851456/3.bitcask.data, 14
41394, /var/lib/riak/bitcask/783558983295571974700998997227772829874127896576/3.bitcask.data, 14
Fix log spam introduced by branch 'gh62-badrecord-mstate'
So, I worried a bit about the possibility of spamminess of
the changes made by branch 'gh62-badrecord-mstate' at the time
I wrote it, but not enough.  In a pull-the-power-cord failure,
the next Riak start included tens of thousands of error messages.
See example below.

Fix: Introduce a cutoff of 20 CRC errors when folding over any
single file.

    ( echo "line-count path bytes-skipped" ; grep 'skip' ALL.console.log | awk '{print $11, $15}' | uniq -c | sort -k 2 | sed 's/^  *//' ) | sed 's/ /, /g'
    42918, /var/lib/riak/bitcask/1019054852592055355075616182186939527377281089536/3.bitcask.data, 14
    1, /var/lib/riak/bitcask/1183188337253240741397319068673631468061296951296/3.bitcask.data, 1049176
    34757, /var/lib/riak/bitcask/1218869529570889738423776217909868846470865616896/3.bitcask.data, 14
    23, /var/lib/riak/bitcask/1247414483425008936044941937298858749198520549376/3.bitcask.data, 14
    1, /var/lib/riak/bitcask/1283095675742657933071399086535096127608089214976/3.bitcask.data, 1049177
    23444, /var/lib/riak/bitcask/1283095675742657933071399086535096127608089214976/3.bitcask.data, 14
    1, /var/lib/riak/bitcask/1290231914206187732476690516382343603290002948096/3.bitcask.data, 1049177
    36198, /var/lib/riak/bitcask/1290231914206187732476690516382343603290002948096/3.bitcask.data, 14
    292, /var/lib/riak/bitcask/248341098530837019304141758684212153730597912576/3.bitcask.data, 14
    258, /var/lib/riak/bitcask/398202106264962806815261785476409143050786308096/3.bitcask.data, 14
    165, /var/lib/riak/bitcask/419610821655552205031136075018151570096527507456/3.bitcask.data, 14
    156, /var/lib/riak/bitcask/433883298582611803841718934712646521460354973696/3.bitcask.data, 14
    1, /var/lib/riak/bitcask/533790637072028995515798952574111181007147237376/3.bitcask.data, 1049176
    3954, /var/lib/riak/bitcask/533790637072028995515798952574111181007147237376/3.bitcask.data, 14
    1, /var/lib/riak/bitcask/626561737097916387784587540588328364872025767936/3.bitcask.data, 1049174
    77531, /var/lib/riak/bitcask/62798898479062234766564582655777786000840851456/3.bitcask.data, 14
    41394, /var/lib/riak/bitcask/783558983295571974700998997227772829874127896576/3.bitcask.data, 14
+fold_loop(_Fd, Filename, _FTStamp, _Header, Offset, _Fun, Acc, 20) ->
+ error_logger:error_msg("fold_loop: CRC error limit at file ~p offset ~p\n",
+ [Filename, Offset]),
+ Acc;
@kellymclaughlin

kellymclaughlin Nov 13, 2012

Contributor

It looks like the fold is stopped once the error limit is reached. Why not continue the fold, but just silence the errors after 20?

@slfritchie

slfritchie Nov 14, 2012

Contributor

Because it takes forever to scan through a multi-MB or -GB file searching for data that may or may not ever be there, and to avoid data that looks like an intra-record header but is really part of some value.

@ghost ghost assigned slfritchie Nov 14, 2012

Contributor

kellymclaughlin commented Nov 14, 2012

+1 to merge

slfritchie added a commit that referenced this pull request Nov 14, 2012

Merge pull request #66 from basho/slf-crc-error-spam
Fix log spam introduced by branch 'gh62-badrecord-mstate'

@slfritchie slfritchie merged commit 859013d into master Nov 14, 2012

1 check passed

default The Travis build passed
Details

@engelsanchez engelsanchez deleted the slf-crc-error-spam branch Mar 28, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment