core/fault/stacktrace: Implement compression. #1172
Conversation
ben-clayton
force-pushed
the
stacktrace
branch
from
b363000
to
381f7fc
Compare
core/fault/stacktrace/pack.go
Outdated
| bestValue := v | ||
| bestIndex := 0 | ||
| bestScore := size(uint64(bestIndex)) + size(bestValue) | ||
| for idx := 1; idx < p.history; idx++ { |
There was a problem hiding this comment.
I do not get this. Why not just:
for j := 0; j < i; j++ {
idx := i - j
There was a problem hiding this comment.
The index used to be fixed size - you're right in that the history length is now dynamic, so yes, this is a good suggestion.
| return out | ||
| } | ||
|
|
||
| type compressorList []compressor |
There was a problem hiding this comment.
That seems like unnecessary complexity. Does that gain anything at all?
There was a problem hiding this comment.
Yeah - it lets you combine compressors together. Experimenting with different compressor combinations has been really useful.
core/fault/stacktrace/pack.go
Outdated
| /* 7 */ compressorList{baseCompressor{}, backRefCompressor{packDiff, unpackDiff, 15}}, | ||
| } | ||
|
|
||
| const compressorIdxBits = 4 // Must be large enough to hold the index of packer. |
|
|
||
| // Tweakables | ||
| const ( | ||
| bitChunkMinSize = 2 |
There was a problem hiding this comment.
Notes:
- Just increments might not be optimal. If you build a histogram of the values, you can probably calculate the optimal chunk sizes.
- Different optimal values might apply to the back-refences and to the deltas.
- Are all of the addresses aligned by any chance? (i.e. can we divide all by common factor?)
- Finally, I do not think it is really worth the time to investigate anything I just said :-D
There was a problem hiding this comment.
Just increments might not be optimal. If you build a histogram of the values, you can probably calculate the optimal chunk sizes.
I actually tried this - storing the table outweighed the costs for this case. I was surprised too.
Are all of the addresses aligned by any chance? (i.e. can we divide all by common factor?)
Nope. Again, something that surprised me. My guess is that these aren't real PC addresses, but contain other metadata somehow.
The goal here is to get a stacktrace down to something under a 150 bytes so it could be reported in a Google Analytics exception message. I've implemented a bunch of different compression methods, compared these to flate, gzip, lzw and zlib. All of the custom compressors out-compress the standard compressors. As compression payloads are small, and time isn't that important, the algorithm tries to compress with 8 different compressors - and encodes the compression type as the first 3 bits of the data. Currently unused, but hopefully will be soon.
ben-clayton
force-pushed
the
stacktrace
branch
from
381f7fc
to
df2175d
Compare
Added Power Summary Track and Graph
The goal here is to get a stacktrace down to something under a 150 bytes so it could be reported in a Google Analytics exception message.
I've implemented a bunch of different compression methods, compared these to flate, gzip, lzw and zlib. All of the custom compressors out-compress the standard compressors.
As compression payloads are small, and time isn't that important, the algorithm tries to compress with 8 different compressors - and encodes the compression type as the first 3 bits of the data.
Currently unused, but hopefully will be soon.