Add standalone coordinator benchmarker #654
Conversation
arnikola
force-pushed
the
arnikola/benchmarker
branch
from
8a16c41
to
b59dc2d
Compare
Codecov Report
@@ Coverage Diff @@
## master #654 +/- ##
==========================================
+ Coverage 80.78% 81.29% +0.51%
==========================================
Files 277 276 -1
Lines 24953 24599 -354
==========================================
- Hits 20158 19998 -160
+ Misses 3601 3406 -195
- Partials 1194 1195 +1
Continue to review full report at Codecov.
|
| scanner := bufio.NewScanner(inFile) | ||
| for w := 0; w < workers; w++ { | ||
| for b := 0; b < batchFiles; b++ { | ||
| outFilePath := fmt.Sprintf("%s/%s%d_%d", dir, toFile, w, b) |
There was a problem hiding this comment.
nit: You can use path.Join instead of encoding "/" in the path.
| Timeseries: ts, | ||
| } | ||
| data, _ := proto.Marshal(req) | ||
| return string(snappy.Encode(nil, data)) |
There was a problem hiding this comment.
Probably best to return []byte and outFile.Write(bytes) instead to avoid the large string alloc?
|
|
||
| func main() { | ||
| if cardinality { | ||
| fmt.Println("Calculating cardinality only") |
There was a problem hiding this comment.
Might be best to use loggers everywhere? shrug.
| } | ||
|
|
||
| start := time.Now() | ||
| log.Println("Started benchmark at:", start.Format(time.StampMilli)) |
There was a problem hiding this comment.
Maybe print out how many is going to be written too?
| go func() { | ||
| defer wg.Done() | ||
| for batchNumber := 0; batchNumber < batches; batchNumber++ { | ||
| err = writeToCoordinator(fmt.Sprintf("%s/%s%d_%d", dataDir, dataFile, worker, batchNumber)) |
There was a problem hiding this comment.
nit: Can use path.Join here too.
No description provided.