Reduce build time #1377

ekalinin · 2020-06-20T14:22:25Z

# before changes
$ git stash
$ ./test.sh
....
PASS
ok  	github.com/dgraph-io/badger/v2	802.360s

# took 27m 47s

# with changes
$ git stash apply
$ ./test.sh
...
PASS
ok  	github.com/dgraph-io/badger/v2	541.394s

# took 18m 38s

This change is

CLAassistant · 2020-06-20T14:22:29Z

All committers have signed the CLA.

jarifibrahim

Thanks for fixing this @ekalinin 🎉

Reviewable status: 0 of 1 files reviewed, all discussions resolved (waiting on @ashish-goswami, @jarifibrahim, and @manishrjain)

This PR adds the following changes from **Master branch to the dgraph-maintenance branch**. ``` 25fd0ef Update ristretto to commit f66de99 (#1391) fac972f Update head while replaying value log (#1372) e5fd05a Rework DB.DropPrefix (#1381) adeb842 Fix assert in background compression and encryption. (#1366) ``` The **master and dgraph-maintenace branch** have following difference `+` (in green) means commit missing on dgraph-maintenance `-` (in red) means the commit exists. ```diff + 079f5ae DefaultOptions: Set KeepL0InMemory to false (#1345) + 7e19cac Add immudb to the project list (#1341) + da80eb9 Iterator: Always add key to txn.reads (#1328) + a7e239e StreamWriter: Close head writer (#1347) + 543f353 Fix build on golang tip (#1355) + fd89894 Compaction: Expired keys and delete markers are never purged (#1354) + 056d859 Support disabling conflict detection (#1344) + b762832 Tests: Do not leave behind state goroutines (#1349) + b2267c2 Restore: Account for value size as well (#1358) + 14386ac GC: Consider size of value while rewriting (#1357) + c45d966 Fix assert in background compression and encryption. (#1366) + dd332b0 Avoid panic in filltables() (#1365) + 3f4761d Force KeepL0InMemory to be true when InMemory is true (#1375) + d37ce36 Tests: Use t.Parallel in TestIteratePrefix tests (#1377) + 158d927 Remove second initialization of writech in Open (#1382) + 675efcd Increase default valueThreshold from 32B to 1KB (#1346) + 3042e37 pre allocate cache key for the block cache and the bloom filter cache (#1371) + e013bfd Rework DB.DropPrefix (#1381) + 509de73 Update head while replaying value log (#1372) + 09dfa66 Update ristretto to commit f66de99 (#1391) + 717b89c Enable cross-compiled 32bit tests on TravisCI (#1392) ```  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/dgraph-io/badger/1398)

Also, disable conflict detection in badger to save memory. ``` 0dfb8b4 Changelog for v20.07.0 (dgraph-io/badger#1411) 03ba278 Add missing changelog for v2.0.3 (dgraph-io/badger#1410) 6001230 Revert "Compress/Encrypt Blocks in the background (dgraph-io/badger#1227)" (dgraph-io/badger#1409) 800305e Revert "Buffer pool for decompression (dgraph-io/badger#1308)" (dgraph-io/badger#1408) 63d9309 Revert "fix: Fix race condition in block.incRef (dgraph-io/badger#1337)" (dgraph-io/badger#1407) e0d058c Revert "add assert to check integer overflow for table size (dgraph-io/badger#1402)" (dgraph-io/badger#1406) d981f47 return error if the vlog writes exceeds more that 4GB. (dgraph-io/badger#1400) 7f4e4b5 add assert to check integer overflow for table size (dgraph-io/badger#1402) 8e896a7 Add a contribution guide (dgraph-io/badger#1379) b79aeef Avoid panic on multiple closer.Signal calls (dgraph-io/badger#1401) 717b89c Enable cross-compiled 32bit tests on TravisCI (dgraph-io/badger#1392) 09dfa66 Update ristretto to commit f66de99 (dgraph-io/badger#1391) 509de73 Update head while replaying value log (dgraph-io/badger#1372) e013bfd Rework DB.DropPrefix (dgraph-io/badger#1381) 3042e37 pre allocate cache key for the block cache and the bloom filter cache (dgraph-io/badger#1371) 675efcd Increase default valueThreshold from 32B to 1KB (dgraph-io/badger#1346) 158d927 Remove second initialization of writech in Open (dgraph-io/badger#1382) d37ce36 Tests: Use t.Parallel in TestIteratePrefix tests (dgraph-io/badger#1377) 3f4761d Force KeepL0InMemory to be true when InMemory is true (dgraph-io/badger#1375) dd332b0 Avoid panic in filltables() (dgraph-io/badger#1365) c45d966 Fix assert in background compression and encryption. (dgraph-io/badger#1366) ```

This commit brings the following new changes from badger. This commit also disable conflict detection in badger to save memory. ``` Fix assert in background compression and encryption. (dgraph-io/badger#1366) Avoid panic in filltables() (dgraph-io/badger#1365) Force KeepL0InMemory to be true when InMemory is true (dgraph-io/badger#1375) Tests: Use t.Parallel in TestIteratePrefix tests (dgraph-io/badger#1377) Remove second initialization of writech in Open (dgraph-io/badger#1382) Increase default valueThreshold from 32B to 1KB (dgraph-io/badger#1346) pre allocate cache key for the block cache and the bloom filter cache (dgraph-io/badger#1371) Rework DB.DropPrefix (dgraph-io/badger#1381) Update head while replaying value log (dgraph-io/badger#1372) Update ristretto to commit f66de99 (dgraph-io/badger#1391) Enable cross-compiled 32bit tests on TravisCI (dgraph-io/badger#1392) Avoid panic on multiple closer.Signal calls (dgraph-io/badger#1401) Add a contribution guide (dgraph-io/badger#1379) add assert to check integer overflow for table size (dgraph-io/badger#1402) return error if the vlog writes exceeds more that 4GB. (dgraph-io/badger#1400) Revert "add assert to check integer overflow for table size (dgraph-io/badger#1402)" (dgraph-io/badger#1406) Revert "fix: Fix race condition in block.incRef (dgraph-io/badger#1337)" (dgraph-io/badger#1407) Revert "Buffer pool for decompression (dgraph-io/badger#1308)" (dgraph-io/badger#1408) Revert "Compress/Encrypt Blocks in the background (dgraph-io/badger#1227)" (dgraph-io/badger#1409) Add missing changelog for v2.0.3 (dgraph-io/badger#1410) Changelog for v20.07.0 (dgraph-io/badger#1411) ```

This commit brings following new changes from badger This commit also disable conflict detection in badger to save memory. ``` Fix assert in background compression and encryption. (dgraph-io/badger#1366) Avoid panic in filltables() (dgraph-io/badger#1365) Force KeepL0InMemory to be true when InMemory is true (dgraph-io/badger#1375) Tests: Use t.Parallel in TestIteratePrefix tests (dgraph-io/badger#1377) Remove second initialization of writech in Open (dgraph-io/badger#1382) Increase default valueThreshold from 32B to 1KB (dgraph-io/badger#1346) pre allocate cache key for the block cache and the bloom filter cache (dgraph-io/badger#1371) Rework DB.DropPrefix (dgraph-io/badger#1381) Update head while replaying value log (dgraph-io/badger#1372) Update ristretto to commit f66de99 (dgraph-io/badger#1391) Enable cross-compiled 32bit tests on TravisCI (dgraph-io/badger#1392) Avoid panic on multiple closer.Signal calls (dgraph-io/badger#1401) Add a contribution guide (dgraph-io/badger#1379) add assert to check integer overflow for table size (dgraph-io/badger#1402) return error if the vlog writes exceeds more that 4GB. (dgraph-io/badger#1400) Revert "add assert to check integer overflow for table size (dgraph-io/badger#1402)" (dgraph-io/badger#1406) Revert "fix: Fix race condition in block.incRef (dgraph-io/badger#1337)" (dgraph-io/badger#1407) Revert "Buffer pool for decompression (dgraph-io/badger#1308)" (dgraph-io/badger#1408) Revert "Compress/Encrypt Blocks in the background (dgraph-io/badger#1227)" (dgraph-io/badger#1409) Add missing changelog for v2.0.3 (dgraph-io/badger#1410) Changelog for v20.07.0 (dgraph-io/badger#1411) ```

* Update badger to v20.07.0-rc1 This commit brings following new changes from badger This commit also disable conflict detection in badger to save memory. ``` Fix assert in background compression and encryption. (dgraph-io/badger#1366) Avoid panic in filltables() (dgraph-io/badger#1365) Force KeepL0InMemory to be true when InMemory is true (dgraph-io/badger#1375) Tests: Use t.Parallel in TestIteratePrefix tests (dgraph-io/badger#1377) Remove second initialization of writech in Open (dgraph-io/badger#1382) Increase default valueThreshold from 32B to 1KB (dgraph-io/badger#1346) pre allocate cache key for the block cache and the bloom filter cache (dgraph-io/badger#1371) Rework DB.DropPrefix (dgraph-io/badger#1381) Update head while replaying value log (dgraph-io/badger#1372) Update ristretto to commit f66de99 (dgraph-io/badger#1391) Enable cross-compiled 32bit tests on TravisCI (dgraph-io/badger#1392) Avoid panic on multiple closer.Signal calls (dgraph-io/badger#1401) Add a contribution guide (dgraph-io/badger#1379) add assert to check integer overflow for table size (dgraph-io/badger#1402) return error if the vlog writes exceeds more that 4GB. (dgraph-io/badger#1400) Revert "add assert to check integer overflow for table size (dgraph-io/badger#1402)" (dgraph-io/badger#1406) Revert "fix: Fix race condition in block.incRef (dgraph-io/badger#1337)" (dgraph-io/badger#1407) Revert "Buffer pool for decompression (dgraph-io/badger#1308)" (dgraph-io/badger#1408) Revert "Compress/Encrypt Blocks in the background (dgraph-io/badger#1227)" (dgraph-io/badger#1409) Add missing changelog for v2.0.3 (dgraph-io/badger#1410) Changelog for v20.07.0 (dgraph-io/badger#1411) ``` * Remove unnecessary detect conflicts

* Update badger to v20.07.0-rc1 This commit brings the following new changes from badger. This commit also disable conflict detection in badger to save memory. ``` Fix assert in background compression and encryption. (dgraph-io/badger#1366) Avoid panic in filltables() (dgraph-io/badger#1365) Force KeepL0InMemory to be true when InMemory is true (dgraph-io/badger#1375) Tests: Use t.Parallel in TestIteratePrefix tests (dgraph-io/badger#1377) Remove second initialization of writech in Open (dgraph-io/badger#1382) Increase default valueThreshold from 32B to 1KB (dgraph-io/badger#1346) pre allocate cache key for the block cache and the bloom filter cache (dgraph-io/badger#1371) Rework DB.DropPrefix (dgraph-io/badger#1381) Update head while replaying value log (dgraph-io/badger#1372) Update ristretto to commit f66de99 (dgraph-io/badger#1391) Enable cross-compiled 32bit tests on TravisCI (dgraph-io/badger#1392) Avoid panic on multiple closer.Signal calls (dgraph-io/badger#1401) Add a contribution guide (dgraph-io/badger#1379) add assert to check integer overflow for table size (dgraph-io/badger#1402) return error if the vlog writes exceeds more that 4GB. (dgraph-io/badger#1400) Revert "add assert to check integer overflow for table size (dgraph-io/badger#1402)" (dgraph-io/badger#1406) Revert "fix: Fix race condition in block.incRef (dgraph-io/badger#1337)" (dgraph-io/badger#1407) Revert "Buffer pool for decompression (dgraph-io/badger#1308)" (dgraph-io/badger#1408) Revert "Compress/Encrypt Blocks in the background (dgraph-io/badger#1227)" (dgraph-io/badger#1409) Add missing changelog for v2.0.3 (dgraph-io/badger#1410) Changelog for v20.07.0 (dgraph-io/badger#1411) ``` * Remove unnecessary detect conflicts

* Update badger to v20.07.0-rc1 This commit brings following new changes from badger This commit also disable conflict detection in badger to save memory. ``` Fix assert in background compression and encryption. (dgraph-io/badger#1366) Avoid panic in filltables() (dgraph-io/badger#1365) Force KeepL0InMemory to be true when InMemory is true (dgraph-io/badger#1375) Tests: Use t.Parallel in TestIteratePrefix tests (dgraph-io/badger#1377) Remove second initialization of writech in Open (dgraph-io/badger#1382) Increase default valueThreshold from 32B to 1KB (dgraph-io/badger#1346) pre allocate cache key for the block cache and the bloom filter cache (dgraph-io/badger#1371) Rework DB.DropPrefix (dgraph-io/badger#1381) Update head while replaying value log (dgraph-io/badger#1372) Update ristretto to commit f66de99 (dgraph-io/badger#1391) Enable cross-compiled 32bit tests on TravisCI (dgraph-io/badger#1392) Avoid panic on multiple closer.Signal calls (dgraph-io/badger#1401) Add a contribution guide (dgraph-io/badger#1379) add assert to check integer overflow for table size (dgraph-io/badger#1402) return error if the vlog writes exceeds more that 4GB. (dgraph-io/badger#1400) Revert "add assert to check integer overflow for table size (dgraph-io/badger#1402)" (dgraph-io/badger#1406) Revert "fix: Fix race condition in block.incRef (dgraph-io/badger#1337)" (dgraph-io/badger#1407) Revert "Buffer pool for decompression (dgraph-io/badger#1308)" (dgraph-io/badger#1408) Revert "Compress/Encrypt Blocks in the background (dgraph-io/badger#1227)" (dgraph-io/badger#1409) Add missing changelog for v2.0.3 (dgraph-io/badger#1410) Changelog for v20.07.0 (dgraph-io/badger#1411) ``` * Remove unnecessary detect conflicts Co-authored-by: parasssh <paras@dgraph.io>

* Update badger to v20.0.7-rc1 This commit brings following new changes from badger This commit also disable conflict detection in badger to save memory. ``` Fix assert in background compression and encryption. (dgraph-io/badger#1366) Avoid panic in filltables() (dgraph-io/badger#1365) Force KeepL0InMemory to be true when InMemory is true (dgraph-io/badger#1375) Tests: Use t.Parallel in TestIteratePrefix tests (dgraph-io/badger#1377) Remove second initialization of writech in Open (dgraph-io/badger#1382) Increase default valueThreshold from 32B to 1KB (dgraph-io/badger#1346) pre allocate cache key for the block cache and the bloom filter cache (dgraph-io/badger#1371) Rework DB.DropPrefix (dgraph-io/badger#1381) Update head while replaying value log (dgraph-io/badger#1372) Update ristretto to commit f66de99 (dgraph-io/badger#1391) Enable cross-compiled 32bit tests on TravisCI (dgraph-io/badger#1392) Avoid panic on multiple closer.Signal calls (dgraph-io/badger#1401) Add a contribution guide (dgraph-io/badger#1379) add assert to check integer overflow for table size (dgraph-io/badger#1402) return error if the vlog writes exceeds more that 4GB. (dgraph-io/badger#1400) Revert "add assert to check integer overflow for table size (dgraph-io/badger#1402)" (dgraph-io/badger#1406) Revert "fix: Fix race condition in block.incRef (dgraph-io/badger#1337)" (dgraph-io/badger#1407) Revert "Buffer pool for decompression (dgraph-io/badger#1308)" (dgraph-io/badger#1408) Revert "Compress/Encrypt Blocks in the background (dgraph-io/badger#1227)" (dgraph-io/badger#1409) Add missing changelog for v2.0.3 (dgraph-io/badger#1410) Changelog for v20.07.0 (dgraph-io/badger#1411) ``` * Remove unnecessary detect conflicts Co-authored-by: parasssh <paras@dgraph.io>

* Update badger to v20.07.0-rc1 This commit brings the following new changes from badger. This commit also disable conflict detection in badger to save memory. ``` Fix assert in background compression and encryption. (dgraph-io/badger#1366) Avoid panic in filltables() (dgraph-io/badger#1365) Force KeepL0InMemory to be true when InMemory is true (dgraph-io/badger#1375) Tests: Use t.Parallel in TestIteratePrefix tests (dgraph-io/badger#1377) Remove second initialization of writech in Open (dgraph-io/badger#1382) Increase default valueThreshold from 32B to 1KB (dgraph-io/badger#1346) pre allocate cache key for the block cache and the bloom filter cache (dgraph-io/badger#1371) Rework DB.DropPrefix (dgraph-io/badger#1381) Update head while replaying value log (dgraph-io/badger#1372) Update ristretto to commit f66de99 (dgraph-io/badger#1391) Enable cross-compiled 32bit tests on TravisCI (dgraph-io/badger#1392) Avoid panic on multiple closer.Signal calls (dgraph-io/badger#1401) Add a contribution guide (dgraph-io/badger#1379) add assert to check integer overflow for table size (dgraph-io/badger#1402) return error if the vlog writes exceeds more that 4GB. (dgraph-io/badger#1400) Revert "add assert to check integer overflow for table size (dgraph-io/badger#1402)" (dgraph-io/badger#1406) Revert "fix: Fix race condition in block.incRef (dgraph-io/badger#1337)" (dgraph-io/badger#1407) Revert "Buffer pool for decompression (dgraph-io/badger#1308)" (dgraph-io/badger#1408) Revert "Compress/Encrypt Blocks in the background (dgraph-io/badger#1227)" (dgraph-io/badger#1409) Add missing changelog for v2.0.3 (dgraph-io/badger#1410) Changelog for v20.07.0 (dgraph-io/badger#1411) ``` * Remove unnecessary detect conflicts

Fixes #1376 # before changes $ git stash $ ./test.sh .... PASS ok github.com/dgraph-io/badger/v2 802.360s # took 27m 47s # with changes $ git stash apply $ ./test.sh ... PASS ok github.com/dgraph-io/badger/v2 541.394s # took 18m 38s

reduce build time

2d9e3e2

ekalinin requested review from ashish-goswami, jarifibrahim and manishrjain as code owners June 20, 2020 14:22

jarifibrahim approved these changes Jun 20, 2020

View reviewed changes

jarifibrahim merged commit d37ce36 into dgraph-io:master Jun 20, 2020

ekalinin deleted the reduce-build-time branch June 20, 2020 16:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce build time #1377

Reduce build time #1377

ekalinin commented Jun 20, 2020 •

edited

CLAassistant commented Jun 20, 2020 •

edited

jarifibrahim left a comment

Reduce build time #1377

Reduce build time #1377

Conversation

ekalinin commented Jun 20, 2020 • edited

CLAassistant commented Jun 20, 2020 • edited

jarifibrahim left a comment

Choose a reason for hiding this comment

ekalinin commented Jun 20, 2020 •

edited

CLAassistant commented Jun 20, 2020 •

edited