-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: reduce memory held by deferred objects #96
Conversation
`bytes.Buffer` will always overallocate (for performance) and we don't have a great way of predicting the correct size. So, instead, copy once when we're done.
7b3233f
to
c98f5cc
Compare
// Reuse any existing buffers. | ||
reusedBuf := d.Raw[:0] | ||
d.Raw = nil | ||
buf := bytes.NewBuffer(reusedBuf) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Checking for understanding:
In the old code reusedBuf
was overallocated compared to the data in d.Raw
, i.e. capacity was greater than length. When we assign reusedBuf
to the slice of Raw
the over capacity is carried along with it. Then when we pass reusedBuf
to bytes.NewBuffer
it further overallocates more capacity than is needed by reusedBuf
, amplifying the allocation further.
Is this correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly the second part. In the case we care about, reusedBuf
was always empty. But bytes.Buffer
generally over-allocates for better performance.
// Reuse existing buffer. Also, copy to "give back" the allocation in the byte buffer (which | ||
// is likely significant). | ||
d.Raw = d.Raw[:0] | ||
d.Raw = append(d.Raw, buf.Bytes()...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Checking for understanding:
In the old code the assignment of buf.Bytes()
to d.Raw
brought along the overallocated capacity. Now by assigning d.Raw
to the append
call this is no longer happening. d.Raw
keeps its old capacity and the append call will only increase its capacity if the length of buf.Bytes()
exceeds capacity.
Is this correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, no. I'm keeping any overallocation from Raw
(reusing the buffer) but removing any overallocation from the bytes Buffer by copying.
Note: in most cases, d.Raw
will be nil. It won't be nil in some special cases where, e.g., we're re-using the same one in a loop. In that case, we don't care about a bit of overallocation as we're re-using space.
The issue with bytes.Buffer
is that we're over-allocating entirely new space.
to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM aside from a coupe of minor comments. It would be great to add some benchmarks to this repo, specially for memory sensitive parts.
Question:
Why not
- pool
Deferred
, - use
bytes.Buffer
in it instead ofRaw []byte
directly, - and handle
pool.Put
withinClose() error
by makingDeferred
anio.Closer
.
?
If I have not missed anything that should remove the need for copy here and make the the buffer/deferred management more efficient to manage. A user would simply borrow Deferred
closers and close them off when done.
defer func() { | ||
buf.Reset() | ||
deferredBufferPool.Put(buf) | ||
}() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
WriteMajorTypeHeaderBuf
taking buf
seem to know precisely how many bytes are getting written. Ideally, we should grow the buffer before writing considering that function is called in a loop.
But doing that needs a bit more refactoring which seems to be beyond the scope of this PR. Please feel free to address this in a separate PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bytes.Buffer
will expand by, IIRC, at least 512 bytes each time, so it shouldn't really matter.
Co-authored-by: Masih H. Derkani <m@derkani.org>
We give We could put a
|
to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90
* deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi <orjan.roren@gmail.com> Co-authored-by: Steven Allen <steven@stebalien.com>
* deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com>
* deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com>
* Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <forcedebug@outlook.com> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * #11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from #11723 Closes: #11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <forcedebug@outlook.com> Co-authored-by: parthshah1 <parthhshah171@gmail.com> Co-authored-by: Piotr Galar <piotr.galar@gmail.com> Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Masih H. Derkani <m@derkani.org> Co-authored-by: Lee <octalgah@gmail.com> Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com> Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
* fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * 1 * relatable * add and delete layer * chore: curio: merge master (#11956) * Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <forcedebug@outlook.com> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * #11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from #11723 Closes: #11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <forcedebug@outlook.com> Co-authored-by: parthshah1 <parthhshah171@gmail.com> Co-authored-by: Piotr Galar <piotr.galar@gmail.com> Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Masih H. Derkani <m@derkani.org> Co-authored-by: Lee <octalgah@gmail.com> Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com> Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com> * linter oops * gen cleanup * fix * named returns are confusing --------- Signed-off-by: forcedebug <forcedebug@outlook.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com> Co-authored-by: parthshah1 <parthhshah171@gmail.com> Co-authored-by: Piotr Galar <piotr.galar@gmail.com> Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Masih H. Derkani <m@derkani.org> Co-authored-by: Lee <octalgah@gmail.com> Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
* Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <forcedebug@outlook.com> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * filecoin-project/lotus#11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from filecoin-project/lotus#11723 Closes: filecoin-project/lotus#11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <forcedebug@outlook.com> Co-authored-by: parthshah1 <parthhshah171@gmail.com> Co-authored-by: Piotr Galar <piotr.galar@gmail.com> Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Masih H. Derkani <m@derkani.org> Co-authored-by: Lee <octalgah@gmail.com> Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com> Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
* Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <forcedebug@outlook.com> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * filecoin-project/lotus#11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from filecoin-project/lotus#11723 Closes: filecoin-project/lotus#11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <forcedebug@outlook.com> Co-authored-by: parthshah1 <parthhshah171@gmail.com> Co-authored-by: Piotr Galar <piotr.galar@gmail.com> Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Masih H. Derkani <m@derkani.org> Co-authored-by: Lee <octalgah@gmail.com> Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com> Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
* fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * 1 * relatable * add and delete layer * chore: curio: merge master (#11956) * Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <forcedebug@outlook.com> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * filecoin-project/lotus#11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from filecoin-project/lotus#11723 Closes: filecoin-project/lotus#11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <forcedebug@outlook.com> Co-authored-by: parthshah1 <parthhshah171@gmail.com> Co-authored-by: Piotr Galar <piotr.galar@gmail.com> Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Masih H. Derkani <m@derkani.org> Co-authored-by: Lee <octalgah@gmail.com> Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com> Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com> * linter oops * gen cleanup * fix * named returns are confusing --------- Signed-off-by: forcedebug <forcedebug@outlook.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com> Co-authored-by: parthshah1 <parthhshah171@gmail.com> Co-authored-by: Piotr Galar <piotr.galar@gmail.com> Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Masih H. Derkani <m@derkani.org> Co-authored-by: Lee <octalgah@gmail.com> Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
* Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <forcedebug@outlook.com> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * filecoin-project/lotus#11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from filecoin-project/lotus#11723 Closes: filecoin-project/lotus#11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <forcedebug@outlook.com> Co-authored-by: parthshah1 <parthhshah171@gmail.com> Co-authored-by: Piotr Galar <piotr.galar@gmail.com> Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Masih H. Derkani <m@derkani.org> Co-authored-by: Lee <octalgah@gmail.com> Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com> Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
* release: v1.26.3 (#11908) (#11915) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> * build: release: v1.27.0-rc1 (#11947) * chore: Set version as v1.27.0-rc1 Set version as v1.27.0-rc1, run make gen & make docsgen-cli * Update changelog Update changelog * Update changelog Update changelog based on feedback * Bump pubsub-dep Bump pubsub-dep * Prep v1.27.0-rc2 Prep v1.27.0-rc2 * Typo fixes, and more changelog updates Typo fixes, and more changelog updates * chore: remove unmaintained bootstrappers (#11983) * chore: remove unmaintained bootstrappers chore: remove unmaintained bootstrappers * Update mainnet.pi fixing typoed domain fixing typo for 1475.io 'bootstarp' -> 'bootstrap' * Update mainnet.pi apparently the actual hostname is typoed. so bootstarp it is. --------- Co-authored-by: smagdali <stefan@fil.org> * chore: update go-data-transfer and go-graphsync * add ETH addrs API to Gateway (#11979) * fix: copy Flags field from SectorOnChainInfo Fixes: #11962 * feat: libp2p: Lotus stream cleanup (#11993) * set stream deadlines in Lotus * reduce timeout * whitelist bootstrappers * fix tests * Update changelog and version Update changelog and version * ci: deprecate circle ci in favour of github actions (#11786) * Update changelog Update changelog with the deprecate circle-ci * chore: update drand (#12021) * Update changelog / make docsgen Update changelog / make docsgen * chore: lint: update golangci lint config * remove and replace some linters * remove some exclusions * make all exclusions more explicit matches * chore: lint: fix lint errors with new linting config Ref: #11967 * chore: lint: address feedback from reviews * doc: eth: restore comment lost in linter cleanup Ref: #11968 * chore: libp2p: update to v0.34.1 (#12027) * update libp2p to v0.34.0 * fix libp2p err * fix imports * update go mod * update go mod * Update changelog Update changelog * go mod tidy go mod tidy * revert go version change (#12050) * Update changelog Update changelog * chore: backport #12054 to release/v1.27.0 branch (#12056) * chore: pin golanglint-ci to v1.58.2 (#12054) Fixes: #12044 * Add backport to changelog Add backport to changelog --------- Co-authored-by: Rod Vagg <rod@vagg.org> * Fix where #12054 is placed in changelog Fix where #12054 is placed in changelog * Add trailing line in Makefile Add trailing line in Makefile * Go mod tidy Go mod tidy --------- Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: smagdali <stefan@fil.org> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com> Co-authored-by: Piotr Galar <piotr.galar@gmail.com>
bytes.Buffer
will always overallocate (for performance) and we don't have a great way of predicting the correct size. So, instead, copy once when we're done.We we're seeing 10x the memory usage expected in a network migration in Filecoin.