Conversation
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| var ( | ||
| totalKeys int64 | ||
| startTime = time.Now() |
Check warning
Code scanning / CodeQL
Calling the system time Warning
| go func() { | ||
| defer close(chunks) | ||
| for i := 0; ; i++ { | ||
| path := filepath.Join(chunksDir, strconv.Itoa(i)) | ||
| f, err := os.Open(filepath.Clean(path)) | ||
| if err != nil { | ||
| if os.IsNotExist(err) { | ||
| return | ||
| } | ||
| pr, pw := io.Pipe() | ||
| _ = pw.CloseWithError(fmt.Errorf("open chunk %d: %w", i, err)) | ||
| chunks <- pr | ||
| return | ||
| } | ||
| chunks <- f | ||
| } | ||
| }() |
Check notice
Code scanning / CodeQL
Spawning a Go routine Note
| if totalKeys%1_000_000 == 0 { | ||
| elapsed := time.Since(startTime).Seconds() | ||
| fmt.Printf("[Snapshot] keys=%d, keys/sec=%.0f, elapsed=%.2fs\n", | ||
| totalKeys, float64(totalKeys)/elapsed, elapsed) |
Check notice
Code scanning / CodeQL
Floating point arithmetic Note
|
|
||
| elapsed := time.Since(startTime).Seconds() | ||
| fmt.Printf("[Snapshot] Import Done: keys=%d, keys/sec=%.0f, elapsed=%.2fs\n", | ||
| totalKeys, float64(totalKeys)/elapsed, elapsed) |
Check notice
Code scanning / CodeQL
Floating point arithmetic Note
There was a problem hiding this comment.
It's funny, I have almost this exact change set in my branch where I'm working on the crypto transfer simulation. 😅
| importer.AddNode(node) | ||
| if node.Height == 0 { | ||
| totalKeys++ | ||
| if totalKeys%1_000_000 == 0 { |
There was a problem hiding this comment.
Should number of keys before a snapshot be configurable?
There was a problem hiding this comment.
We usually want to finish loading all keys in a snapshot so that we can do hash verification, but I'm open to that feature in a future PR
| func (s *CommitStore) Exporter(version int64) (types.Exporter, error) { | ||
| // TODO: Implement exporter | ||
| return nil, nil | ||
| } |
There was a problem hiding this comment.
Nit, should this panic or return something like fmt.Errof("not implemented")?
There was a problem hiding this comment.
Good call, will do that instead
| type SnapshotImporter struct { | ||
| cosmosImporter types.Importer | ||
| evmImporter types.Importer | ||
| currentModule string |
There was a problem hiding this comment.
it seems currentModule is declared but not set anywhere
| Changeset: iavl.ChangeSet{Pairs: imp.batch}, | ||
| }} | ||
| if err := imp.store.ApplyChangeSets(cs); err != nil { | ||
| imp.err = fmt.Errorf("import apply changesets: %w", err) |
There was a problem hiding this comment.
how about adding some logs here like
imp.store.log.Error("import flush failed, remaining nodes will be dropped", "err", imp.err)
so we can see error log output with context. otherwise we only see errors when Close()
| return | ||
| } | ||
| if err := imp.store.commitBatches(imp.version); err != nil { | ||
| imp.err = fmt.Errorf("import commit batches: %w", err) |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2970 +/- ##
===========================================
+ Coverage 58.15% 61.22% +3.07%
===========================================
Files 2107 16 -2091
Lines 173323 1354 -171969
===========================================
- Hits 100794 829 -99965
+ Misses 63580 403 -63177
+ Partials 8949 122 -8827
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This PR add feature to import snapshot - Add feature for FlatKV import from snapshot - Make go benchmark able to load from local snapshot
* main: (66 commits) feat(flatkv): include legacyDB in ApplyChangeSets, LtHash, and read path (#2978) Deflake mempool tests with Eventually-based block waits (#2983) Demote noisy gasless classification log to debug level (#2982) Harden `TestStateLock_NoPOL` against proposal/timeout race (#2980) added a config parameter to limit outbound p2p connections. (#2974) merged unconditional and persistent peers status (#2977) Fix race between file pruning and in-flight parquet queries (#2975) fix(giga): don't migrate balance on failed txs (#2961) Fix hanging upgrade tests by adding timeouts to wait_for_height (#2976) Add snapshot import for Giga Live State (#2970) Fix Rocksdb MVCC read timestamp lifetime for iterators (#2971) Reduce exposed tendermint RPC endpoint (#2968) Deflake `TestStateLock_NoPOL` by widening propose timeout in test (#2969) go bench read + write receipts/logs for parquet vs pebble (#2794) [giga] clear up cache after Write (#2827) fix: use correct EVM storage key prefix in benchmark key generation (#2966) Harden staking precompile test against CI flakiness (#2967) Don't sync flatKV DBs when committing (#2964) Fix flaky `TestStateLock_POLSafety1` (#2962) Add metrics for historical proof success/failure rate (#2958) ...
3 participants
Describe your changes and provide context
This PR add feature to import snapshot
Testing performed to validate your change