You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added
A reusable memory budget (IMemoryBudget) that work with a large footprint reserves memory from before it runs. Requests are served in order, a request larger than the whole budget runs alone, a wait that exceeds a maximum is let through, and the capacity follows the memory pressure of the process. It is configured under Cratis:Chronicle:Memory (Enabled, BudgetPercentOfAvailableMemory, BudgetPercentUnderPressure, MaximumWaitSeconds, MinimumReservationBytes, ReducerCostMultiplier).
Reduces of large states wait for room in the memory budget, so many large partitions no longer reduce at the same time and exhaust the heap. A reduce reserves for the size its partition's state had last time, and accounts for its state once read when it had no size to go by.
Metrics for how the process uses memory (cratis.chronicle.memory.*: heap size, committed, available, load, fragmentation, pause time ratio, working set, allocated bytes, full collections), for the budget (capacity, reserved, waiting, wait time, queued, overcommitted), and for the estimated size of reducer states (cratis.chronicle.reducer.state_size).
Changed
The state sent to a reducer client is written straight to JSON, and the state it returns is read straight from JSON, without first building a tree of nodes of the whole document. Strings, guids and dates are read and written without a node each, and dictionaries without reflection, which roughly halves what converting a large state allocates and keeps the whole document from being held a second time.
A schema's type is read once instead of on every value converted.
IExpandoObjectConverter gains WriteJson and a ToExpandoObject that reads UTF-8 JSON, both with default implementations, so existing implementations keep working.