feat(pprof): Pprof HTTP server service #1991

qdm12 · 2021-11-05T14:27:12Z

Changes

Add a pprof HTTP server service
- internal/httpserver package
- internal/pprof package
Register this service in the dot node
Configurable listening address
Configurable pprof server on/off
Add config tests
Finish documentation
~~Add copyright notices~~ will do with docs(copyright): Update copyright generation and notice per file #1997
Notes:
- this does not affect performance, except when triggered by an HTTP client
- this listens on localhost:6060 (localhost so it works with IPv6 local too)

Tests

go test -race github.com/ChainSafe/gossamer/internal/pprof
go test -race github.com/ChainSafe/gossamer/internal/httpserver
go test github.com/ChainSafe/gossamer/dot

Issues

Pprof http server #1969

Primary Reviewer

codecov · 2021-11-05T15:44:44Z

Codecov Report

Merging #1991 (ffece83) into development (8c2993d) will increase coverage by 0.18%.
The diff coverage is 67.18%.

@@               Coverage Diff               @@
##           development    #1991      +/-   ##
===============================================
+ Coverage        60.12%   60.30%   +0.18%     
===============================================
  Files              195      200       +5     
  Lines            26671    26803     +132     
===============================================
+ Hits             16035    16164     +129     
+ Misses            8755     8750       -5     
- Partials          1881     1889       +8

Flag	Coverage Δ
unit-tests	`60.30% <67.18%> (+0.18%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
cmd/gossamer/flags.go	`66.66% <ø> (ø)`
cmd/gossamer/main.go	`14.40% <0.00%> (+0.22%)`	⬆️
dot/config.go	`25.89% <21.62%> (-0.74%)`	⬇️
cmd/gossamer/config.go	`74.96% <37.14%> (-2.17%)`	⬇️
internal/pprof/settings.go	`50.00% <50.00%> (ø)`
internal/httpserver/run.go	`83.72% <83.72%> (ø)`
cmd/gossamer/export.go	`88.46% <100.00%> (+0.83%)`	⬆️
cmd/gossamer/utils.go	`74.35% <100.00%> (+0.33%)`	⬆️
dot/node.go	`56.96% <100.00%> (-0.14%)`	⬇️
dot/services.go	`68.60% <100.00%> (+0.30%)`	⬆️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c2993d...ffece83. Read the comment docs.

dot/services.go

internal/httpserver/address.go

internal/httpserver/matchers_test.go

EclesioMeloJunior

Would be nice to use go:generate to generate the regex_matchers_test.go files?

qdm12 · 2021-11-08T15:38:19Z

Would be nice to use go:generate to generate the regex_matchers_test.go files?

These are files manually written 😄 It's just a custom matcher accepting regexes to assert the mocked function arguments. We could put it in test focused internal package later if we feel we need it for other tests. But I also like the "a bit of code duplication is better than dependency headaches" 😉

noot

love the docs!! I have one question, if someone sets --pprofserver, but the block rate and mutex rate are set to 0, will the server run or no?

qdm12 · 2021-11-11T09:23:14Z

@noot Yes the server will run, it's just the /debug/pprof/block and /debug/pprof/mutex routes won't be working. Note the other way around, if you set e.g. --pprofmutexrate 1 without --pprofserver, it won't be set since the server is disabled. Let me know if I should change that/document that.

qdm12 · 2021-11-15T13:34:42Z

I'm thinking of either:

removing the existing profiling functions (to a file in cmd/gossamer/profile.go) in favor of the http server; or
moving the existing profiling functions in the pprof service with additional settings to choose if the program should write to a file and/or run an http server.

I am leaning towards the first one, but I don't want to hurt feelings or habits either 😸 What do you think?

- Call `runtime.SetBlockProfileRate` - Call `runtime.SetMutexProfileFraction` - Parametrable rates

- Read from flags - Read from TOML config file - Update toml files with default values

- because it impacts performance if running all the time - because you can easily download the file from the http server - because it clutters the main code

# [0.6.0](v0.5.0...v0.6.0) (2021-12-03) ### Bug Fixes * **babe:** Fix extrinsic format in block. ([#1530](#1530)) ([1a03b2a](1a03b2a)) * **ci:** add missing go-version matrix to fix development branch CI ([#2037](#2037)) ([6babe76](6babe76)) * **cmd/cfg:** Use Babe Lead value from toml config ([#2032](#2032)) ([06aa3e3](06aa3e3)) * cmd/gossamer: Generate random name if --name flag not set ([#1506](#1506)) ([3c05a88](3c05a88)) * confirm block import notifier is closed properly ([#1736](#1736)) ([ad2d85e](ad2d85e)) * **docs:** improve build-spec usage docs ([#1706](#1706)) ([2e164b4](2e164b4)) * **dot/core:** Add only extrinsic during chain reorg. ([#1609](#1609)) ([29413d4](29413d4)) * **dot/core:** Batch process transaction message. ([#1780](#1780)) ([0064836](0064836)) * **dot/core:** check transaction Validity.Propagate field to determine whether to propagate tx ([#1643](#1643)) ([81f23cc](81f23cc)) * **dot/core:** Fix handle transaction message test. ([#1607](#1607)) ([58b8725](58b8725)) * **dot/network, lib/grandpa:** fix handshake decoding and grandpa message handler sigabort ([#1631](#1631)) ([887f72c](887f72c)) * **dot/network, lib/grandpa:** fix node sync, improve devnet finality ([bcc7935](bcc7935)) * **dot/network:** add nil checks in connManager ([#2069](#2069)) ([7f9c042](7f9c042)) * **dot/network:** Check for size when decoding leb128. ([#1634](#1634)) ([d082b9e](d082b9e)) * **dot/network:** check if peer supports protocol ([#1617](#1617)) ([6bf66a4](6bf66a4)) * **dot/network:** decrease DHT find peers interval for gssmr nodes ([#1703](#1703)) ([08516a0](08516a0)) * **dot/network:** fix bugs in notifications protocol handlers; add metrics for inbound/outbound streams ([#2010](#2010)) ([8c2993d](8c2993d)) * **dot/network:** fix dht connection on discovery on devnet ([#2059](#2059)) ([da065b8](da065b8)) * **dot/network:** fix discovery between gossamer nodes ([#1594](#1594)) ([f4c79d3](f4c79d3)) * **dot/network:** fix justification request at head logic ([#1510](#1510)) ([98d1413](98d1413)) * **dot/network:** fix memory allocations with `sizedBufferPool` ([#1963](#1963)) ([e0b126b](e0b126b)) * **dot/network:** Fix missing digest in header ([#2092](#2092)) ([21ea85e](21ea85e)) * **dot/network:** Fix notification handshake and reuse stream. ([#1545](#1545)) ([a632dc4](a632dc4)) * **dot/network:** fix receiving notifications messages from substrate peers ([#1517](#1517)) ([fdf3c53](fdf3c53)) * **dot/network:** fix stream manager tests ([#1683](#1683)) ([e02eca4](e02eca4)) * **dot/network:** implement a handshake timeout ([#1615](#1615)) ([87c2f63](87c2f63)) * **dot/network:** Implement time based handle transaction ([#1942](#1942)) ([dd08424](dd08424)) * **dot/network:** move low reputation peer removal from network ConnManager to peer scoring logic (dot/peerstate) ([#2068](#2068)) ([ac16285](ac16285)), closes [#2039](#2039) * **dot/network:** Return on EOF error while reading stream. ([#1733](#1733)) ([f447eac](f447eac)) * **dot/network:** split stored streams and handshakeData into inbound and outbound ([#1553](#1553)) ([637050b](637050b)) * **dot/network:** update notificationsProtocol handshakeData to sync.Map ([#1492](#1492)) ([22f7269](22f7269)) * **dot/node:** Start websocket server only with `--ws` flag ([#1671](#1671)) ([6ecef3b](6ecef3b)) * **dot/state, lib/babe, lib/trie:** improve syncing between gossamer authority nodes ([#1613](#1613)) ([ca99fbf](ca99fbf)) * **dot/state, lib/grandpa:** update justification and SignedVote handling in database ([#1682](#1682)) ([bbdcd6f](bbdcd6f)) * **dot/state,dot/network:** improve memory usage when syncing ([#1491](#1491)) ([3b2ad8d](3b2ad8d)) * **dot/state:** add StorageState Lock/Unlock API for usage by babe and sync ([#1700](#1700)) ([3c22ace](3c22ace)) * **dot/state:** fix deadlock, fixes bootstrap syncing ([#1959](#1959)) ([dd80c09](dd80c09)) * **dot/state:** fix usage of trie.Snapshot ([#1489](#1489)) ([3880a40](3880a40)) * **dot/state:** track runtime per-block, fix runtime upgrades differing between forks ([#1638](#1638)) ([e133884](e133884)) * **dot/state:** update `*state.BlockState.AddBlockToBlockTree` to store block in `unfinalisedBlocksMap` ([#2006](#2006)) ([55d997f](55d997f)) * **dot/sync:** add nil header checks ([#2099](#2099)) ([a7d4be0](a7d4be0)) * **dot/sync:** fix block request and response logic ([#1907](#1907)) ([9c6283e](9c6283e)) * **dot/sync:** fix creating block response, fixes node sync between gossamer nodes ([#1572](#1572)) ([1328c80](1328c80)) * **dot/telemetry:** refactor telemetry to reduce CPU usage ([#1597](#1597)) ([bc31ac7](bc31ac7)) * **dot/types:** *types.Body to be of type []types.Extrinsic ([#1807](#1807)) ([4c09715](4c09715)) * **dot/types:** fix max value for digest ([#1687](#1687)) ([48405e7](48405e7)) * **dot:** fix `TestNewNode` ([#2070](#2070)) ([42908d0](42908d0)) * fix edit link ([#1507](#1507)) ([5089327](5089327)) * fix Kusama sync; add storageState lock in core.HandleTransactionMessage ([#1783](#1783)) ([1d688e4](1d688e4)) * **lib/babe, lib/runtime/wasmer:** fixes for v0.9.8+ runtime ([#2075](#2075)) ([2f9f80c](2f9f80c)) * **lib/babe:** add `--babe-lead` flag, update epoch handling logic ([#1895](#1895)) ([7abcce6](7abcce6)) * **lib/babe:** add pre-runtime digest before calling initialize_block ([#1581](#1581)) ([c1b26d3](c1b26d3)) * **lib/babe:** always use 2/3 of slot to produce block, re-add potentially valid txs to queue ([#1679](#1679)) ([cf93ad3](cf93ad3)) * **lib/babe:** call AddBlock in BABE synchronously ([#1585](#1585)) ([86acc43](86acc43)) * **lib/babe:** fix BABE state storing after building block ([#1536](#1536)) ([1a3dea2](1a3dea2)) * **lib/babe:** fix err log ([#1801](#1801)) ([a96f06a](a96f06a)) * **lib/babe:** fix setting first slot of network, fix loading BABE epoch params ([#1640](#1640)) ([5c3dbfe](5c3dbfe)) * **lib/babe:** fix timing for transition between epochs ([#1636](#1636)) ([57027db](57027db)) * **lib/blocktree:** fix blocktree bug ([#2060](#2060)) ([c17b53a](c17b53a)) * **lib/blocktree:** fix potential nil pointer dereference in `HighestCommonAncestor`, core `handleBlocksAsync` ([#1993](#1993)) ([f7f4463](f7f4463)) * **lib/blocktree:** fix setting leaves after blocktree pruning ([#1605](#1605)) ([58c0854](58c0854)) * **lib/blocktree:** removes the inconsistency to choose a deepest leaf ([#2094](#2094)) ([43d68e3](43d68e3)) * **lib/crypto/ed25519:** update ed25519 to use go-schnorrkel bip39 derivation ([#1488](#1488)) ([dfb95d2](dfb95d2)) * **lib/genesis:** Update missing and incorrect fields in genesis file. ([#1681](#1681)) ([8207704](8207704)) * **lib/grandpa:** fix grandpa stall and various bugs ([#1708](#1708)) ([67c93f4](67c93f4)) * **lib/grandpa:** fix grandpa vote message switch ([#2095](#2095)) ([461890c](461890c)) * **lib/grandpa:** fix threshold checking to be strictly greater than 2/3 ([#1891](#1891)) ([66ffe51](66ffe51)) * **lib/grandpa:** use `defaultGrandpaInterval` if not set, fixes error on startup ([#1982](#1982)) ([75627b5](75627b5)) * **lib/runtime/life:** remove import C from life ([#1923](#1923)) ([ed507d2](ed507d2)) * **lib/runtime:** update HOST_API_TEST_RUNTIME_URL to reference specific commit ([#1885](#1885)) ([666ed06](666ed06)) * **log-levels:** do not ignore configuration file log levels ([#2016](#2016)) ([80879b2](80879b2)) * pending bubble hidden after block included ([#1592](#1592)) ([5826322](5826322)) * persist node name ([#1543](#1543)) ([88b88f2](88b88f2)) * **pprof:** only run pprof service if enabled ([#2073](#2073)) ([55669c5](55669c5)) * **release:** Trigger release when pushed to main branch. ([#1566](#1566)) ([d445c97](d445c97)) * **rpc/subscription:** subscribe runtime version notify when version changes ([#1686](#1686)) ([9a76d39](9a76d39)) * Staging CI workflow ([#2034](#2034)) ([84ec792](84ec792)) * **trie:** memory leak fix in `lib/trie` ([#2009](#2009)) ([0ad5eb7](0ad5eb7)) * update deprecated package ([#1603](#1603)) ([f195204](f195204)) * update go-schnorrkel version ([#1557](#1557)) ([b86c7ff](b86c7ff)) * update gssmr genesis to use pallet_babe::SameAuthoritiesForever ([#1696](#1696)) ([fb0a751](fb0a751)) * update HOST_API_TEST_RUNTIME_URL ([#1898](#1898)) ([2ef59a8](2ef59a8)) * **utils:** create a specific folder for database ([#1598](#1598)) ([8c67795](8c67795)) ### Features * add --chain dev option ([#1561](#1561)) ([04a2969](04a2969)) * Add properties and chainId on build-spec command ([#1520](#1520)) ([b18290c](b18290c)) * cmd: implement import-runtime subcommand ([#1483](#1483)) ([d82b2da](d82b2da)) * **cmd/gossamer:** implement --telemetry-url parameter ([#1890](#1890)) ([b202e89](b202e89)), closes [#1502](#1502) * **cmd:** implement offline pruning of state trie ([#1564](#1564)) ([af9c925](af9c925)) * **devnet:** Local Gossamer Devnet ([#2008](#2008)) ([a520001](a520001)) * **dot/network, lib/grandpa:** request justification on receiving NeighbourMessage, verify justification on receipt ([#1529](#1529)) ([e1f9f42](e1f9f42)) * **dot/network:** Add cache for network message. ([#1511](#1511)) ([accaf69](accaf69)) * **dot/network:** add propagate return bool to messageHandler func type to determine whether to propagate message or not ([#1555](#1555)) ([0d6f488](0d6f488)) * **dot/network:** implement persistent peers functionality ([#1512](#1512)) ([7850532](7850532)) * **dot/network:** implement streamManager to cleanup not recently used streams ([#1611](#1611)) ([ba861bf](ba861bf)) * **dot/network:** request block justifications when near head ([#1499](#1499)) ([ae7012b](ae7012b)) * **dot/peerset:** Implement peer scoring ([#1791](#1791)) ([1c989ad](1c989ad)) * **dot/rpc/modules:** add `system_addReservedPeer` and `system_removeReservedPeer` RPC call ([#1712](#1712)) ([dba5922](dba5922)) * **dot/rpc:** Add `system_localListenAddresses` RPC call ([#1689](#1689)) ([c981d2e](c981d2e)) * **dot/rpc:** Implement `childstate_getKeys` rpc call ([#1800](#1800)) ([9b2f41e](9b2f41e)) * **dot/rpc:** implement sync_state_genSyncSpec RPC call ([#1827](#1827)) ([2186caf](2186caf)) * **dot/state:** implement online pruning of historical state tries ([#1596](#1596)) ([3eb9399](3eb9399)) * **dot/sync:** implement codeSubstitutes ([#1635](#1635)) ([d87aaeb](d87aaeb)) * dot/telemetry: Implement basic telemetry connection ([#1497](#1497)) ([fcb4159](fcb4159)) * **dot/telemetry:** Added connection retry ([#1904](#1904)) ([579a791](579a791)) * **dot/telemetry:** Added more telemetry messages in grandpa client ([#2043](#2043)) ([2e57d15](2e57d15)), closes [#1841](#1841) [#1842](#1842) * **dot/telemetry:** implement notify.finalized telemetry interface ([#1877](#1877)) ([de1a60d](de1a60d)) * **dot/telemetry:** implement substrate_number_leaves metrics ([#1926](#1926)) ([69823c0](69823c0)) * **dot/telemetry:** implement telemetry message network_state ([#1618](#1618)) ([a81844e](a81844e)) * **flags:** read log levels from flags ([#1953](#1953)) ([9694e46](9694e46)) * implement ext_default_child_storage_storage_kill_version_2 ([#1799](#1799)) ([c2908ae](c2908ae)) * implement ext_offchain_index_set_version_1 for wasmer runtime ([#1739](#1739)) ([96c30a6](96c30a6)) * **lib/babe:** add check of types.ConfigData.SecondarySlots for disabling secondary verification ([#1910](#1910)) ([cd27ae4](cd27ae4)) * **lib/grandpa:** fully verify justifications using GrandpaState ([#1544](#1544)) ([028d25e](028d25e)) * **lib/grandpa:** Include equivocatory nodes while creating justification ([#1911](#1911)) ([aca86b6](aca86b6)) * **lib/grandpa:** send NeighbourMessage to peers ([#1558](#1558)) ([322ccf9](322ccf9)) * **lib/runtime/wasmer:** implement ext_default_child_storage_storage_kill_version_3 ([#1878](#1878)) ([a719a60](a719a60)) * **lib/runtime/wasmer:** implement ext_offchain_local_storage_version_1 ([#1821](#1821)) ([0f63b17](0f63b17)) * **lib/runtime:** Implement `ext_offchain_http_request_add_header_version_1` host function ([#1994](#1994)) ([0a30b3d](0a30b3d)) * **lib/runtime:** Implement `ext_offchain_http_request_start_version_1` host function ([#1947](#1947)) ([974b1fc](974b1fc)) * **lib/runtime:** Implement `trie_blake2_256_verify_proof` host function ([#1920](#1920)) ([506565d](506565d)) * **lib/trie:** Implement `verify_proof` function ([#1883](#1883)) ([67bb5ef](67bb5ef)) * **lib/trie:** Implement limit for trie.ClearPrefix ([#1905](#1905)) ([becec9e](becec9e)) * **lib/trie:** Parallel hash trie. ([#1657](#1657)) ([22827e7](22827e7)) * **pprof:** Pprof HTTP server service ([#1991](#1991)) ([ce24ea9](ce24ea9)) * **rpc/subscription:** implement state_unsubscribeStorage ([#1574](#1574)) ([7574f10](7574f10)) * **rpc:** Implement `childstate_getChildStorage` RPC call ([#1832](#1832)) ([3d949f2](3d949f2)) * **rpc:** Implement `childstate_getStorageHash` RPC call ([#1805](#1805)) ([e539bd3](e539bd3)) * **rpc:** Implement `childstate_getStorageSize` RPC call ([#1810](#1810)) ([a04deb6](a04deb6)) * **rpc:** Implement `payment_queryInfo` RPC call ([#1826](#1826)) ([7a5deec](7a5deec)) * **rpc:** Implement `state_getReadProof` rpc call ([#1768](#1768)) ([865f80f](865f80f)) * **runtime:** implement custom logging handler that print function name ([#1825](#1825)) ([2b1276d](2b1276d)) * **telemetry:** send telemetry messages when GRANDPA receieves commit or vote messages ([#2015](#2015)) ([7bf40e1](7bf40e1)), closes [#1840](#1840) [#1839](#1839) [#1838](#1838) * **telemetry:** send txpool.import telemetry msg ([#1966](#1966)) ([ffc81bf](ffc81bf)) ### Reverts * Revert "feat(dot/rpc) implement `author_hasSessionKeys` RPC call (#1704)" (#1714) ([65380fd](65380fd)), closes [#1704](#1704) [#1714](#1714)

github-actions · 2021-12-03T21:04:26Z

🎉 This PR is included in version 0.6.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

# [0.5.0](v0.4.1...v0.5.0) (2021-12-06) ### Bug Fixes * **babe:** Fix extrinsic format in block. ([ChainSafe#1530](https://github.com/timwu20/gossamer/issues/1530)) ([1a03b2a](1a03b2a)) * **ci:** add missing go-version matrix to fix development branch CI ([ChainSafe#2037](https://github.com/timwu20/gossamer/issues/2037)) ([6babe76](6babe76)) * **cmd/cfg:** Use Babe Lead value from toml config ([ChainSafe#2032](https://github.com/timwu20/gossamer/issues/2032)) ([06aa3e3](06aa3e3)) * confirm block import notifier is closed properly ([ChainSafe#1736](https://github.com/timwu20/gossamer/issues/1736)) ([ad2d85e](ad2d85e)) * **docs:** improve build-spec usage docs ([ChainSafe#1706](https://github.com/timwu20/gossamer/issues/1706)) ([2e164b4](2e164b4)) * **dot/core:** Add only extrinsic during chain reorg. ([ChainSafe#1609](https://github.com/timwu20/gossamer/issues/1609)) ([29413d4](29413d4)) * **dot/core:** Batch process transaction message. ([ChainSafe#1780](https://github.com/timwu20/gossamer/issues/1780)) ([0064836](0064836)) * **dot/core:** check transaction Validity.Propagate field to determine whether to propagate tx ([ChainSafe#1643](https://github.com/timwu20/gossamer/issues/1643)) ([81f23cc](81f23cc)) * **dot/core:** Fix handle transaction message test. ([ChainSafe#1607](https://github.com/timwu20/gossamer/issues/1607)) ([58b8725](58b8725)) * **dot/network, lib/grandpa:** fix handshake decoding and grandpa message handler sigabort ([ChainSafe#1631](https://github.com/timwu20/gossamer/issues/1631)) ([887f72c](887f72c)) * **dot/network, lib/grandpa:** fix node sync, improve devnet finality ([bcc7935](bcc7935)) * **dot/network:** add nil checks in connManager ([ChainSafe#2069](https://github.com/timwu20/gossamer/issues/2069)) ([7f9c042](7f9c042)) * **dot/network:** Check for size when decoding leb128. ([ChainSafe#1634](https://github.com/timwu20/gossamer/issues/1634)) ([d082b9e](d082b9e)) * **dot/network:** check if peer supports protocol ([ChainSafe#1617](https://github.com/timwu20/gossamer/issues/1617)) ([6bf66a4](6bf66a4)) * **dot/network:** decrease DHT find peers interval for gssmr nodes ([ChainSafe#1703](https://github.com/timwu20/gossamer/issues/1703)) ([08516a0](08516a0)) * **dot/network:** fix bugs in notifications protocol handlers; add metrics for inbound/outbound streams ([ChainSafe#2010](https://github.com/timwu20/gossamer/issues/2010)) ([8c2993d](8c2993d)) * **dot/network:** fix dht connection on discovery on devnet ([ChainSafe#2059](https://github.com/timwu20/gossamer/issues/2059)) ([da065b8](da065b8)) * **dot/network:** fix discovery between gossamer nodes ([ChainSafe#1594](https://github.com/timwu20/gossamer/issues/1594)) ([f4c79d3](f4c79d3)) * **dot/network:** fix memory allocations with `sizedBufferPool` ([ChainSafe#1963](https://github.com/timwu20/gossamer/issues/1963)) ([e0b126b](e0b126b)) * **dot/network:** Fix missing digest in header ([ChainSafe#2092](https://github.com/timwu20/gossamer/issues/2092)) ([21ea85e](21ea85e)) * **dot/network:** Fix notification handshake and reuse stream. ([ChainSafe#1545](https://github.com/timwu20/gossamer/issues/1545)) ([a632dc4](a632dc4)) * **dot/network:** fix stream manager tests ([ChainSafe#1683](https://github.com/timwu20/gossamer/issues/1683)) ([e02eca4](e02eca4)) * **dot/network:** implement a handshake timeout ([ChainSafe#1615](https://github.com/timwu20/gossamer/issues/1615)) ([87c2f63](87c2f63)) * **dot/network:** Implement time based handle transaction ([ChainSafe#1942](https://github.com/timwu20/gossamer/issues/1942)) ([dd08424](dd08424)) * **dot/network:** move low reputation peer removal from network ConnManager to peer scoring logic (dot/peerstate) ([ChainSafe#2068](https://github.com/timwu20/gossamer/issues/2068)) ([ac16285](ac16285)), closes [ChainSafe#2039](https://github.com/timwu20/gossamer/issues/2039) * **dot/network:** Return on EOF error while reading stream. ([ChainSafe#1733](https://github.com/timwu20/gossamer/issues/1733)) ([f447eac](f447eac)) * **dot/network:** split stored streams and handshakeData into inbound and outbound ([ChainSafe#1553](https://github.com/timwu20/gossamer/issues/1553)) ([637050b](637050b)) * **dot/node:** Start websocket server only with `--ws` flag ([ChainSafe#1671](https://github.com/timwu20/gossamer/issues/1671)) ([6ecef3b](6ecef3b)) * **dot/state, lib/babe, lib/trie:** improve syncing between gossamer authority nodes ([ChainSafe#1613](https://github.com/timwu20/gossamer/issues/1613)) ([ca99fbf](ca99fbf)) * **dot/state, lib/grandpa:** update justification and SignedVote handling in database ([ChainSafe#1682](https://github.com/timwu20/gossamer/issues/1682)) ([bbdcd6f](bbdcd6f)) * **dot/state:** add StorageState Lock/Unlock API for usage by babe and sync ([ChainSafe#1700](https://github.com/timwu20/gossamer/issues/1700)) ([3c22ace](3c22ace)) * **dot/state:** fix deadlock, fixes bootstrap syncing ([ChainSafe#1959](https://github.com/timwu20/gossamer/issues/1959)) ([dd80c09](dd80c09)) * **dot/state:** track runtime per-block, fix runtime upgrades differing between forks ([ChainSafe#1638](https://github.com/timwu20/gossamer/issues/1638)) ([e133884](e133884)) * **dot/state:** update `*state.BlockState.AddBlockToBlockTree` to store block in `unfinalisedBlocksMap` ([ChainSafe#2006](https://github.com/timwu20/gossamer/issues/2006)) ([55d997f](55d997f)) * **dot/sync:** add nil header checks ([ChainSafe#2099](https://github.com/timwu20/gossamer/issues/2099)) ([a7d4be0](a7d4be0)) * **dot/sync:** fix block request and response logic ([ChainSafe#1907](https://github.com/timwu20/gossamer/issues/1907)) ([9c6283e](9c6283e)) * **dot/sync:** fix creating block response, fixes node sync between gossamer nodes ([ChainSafe#1572](https://github.com/timwu20/gossamer/issues/1572)) ([1328c80](1328c80)) * **dot/telemetry:** refactor telemetry to reduce CPU usage ([ChainSafe#1597](https://github.com/timwu20/gossamer/issues/1597)) ([bc31ac7](bc31ac7)) * **dot/types:** *types.Body to be of type []types.Extrinsic ([ChainSafe#1807](https://github.com/timwu20/gossamer/issues/1807)) ([4c09715](4c09715)) * **dot/types:** fix max value for digest ([ChainSafe#1687](https://github.com/timwu20/gossamer/issues/1687)) ([48405e7](48405e7)) * **dot:** fix `TestNewNode` ([ChainSafe#2070](https://github.com/timwu20/gossamer/issues/2070)) ([42908d0](42908d0)) * fix Kusama sync; add storageState lock in core.HandleTransactionMessage ([ChainSafe#1783](https://github.com/timwu20/gossamer/issues/1783)) ([1d688e4](1d688e4)) * **lib/babe, lib/runtime/wasmer:** fixes for v0.9.8+ runtime ([ChainSafe#2075](https://github.com/timwu20/gossamer/issues/2075)) ([2f9f80c](2f9f80c)) * **lib/babe:** add `--babe-lead` flag, update epoch handling logic ([ChainSafe#1895](https://github.com/timwu20/gossamer/issues/1895)) ([7abcce6](7abcce6)) * **lib/babe:** add pre-runtime digest before calling initialize_block ([ChainSafe#1581](https://github.com/timwu20/gossamer/issues/1581)) ([c1b26d3](c1b26d3)) * **lib/babe:** always use 2/3 of slot to produce block, re-add potentially valid txs to queue ([ChainSafe#1679](https://github.com/timwu20/gossamer/issues/1679)) ([cf93ad3](cf93ad3)) * **lib/babe:** call AddBlock in BABE synchronously ([ChainSafe#1585](https://github.com/timwu20/gossamer/issues/1585)) ([86acc43](86acc43)) * **lib/babe:** fix BABE state storing after building block ([ChainSafe#1536](https://github.com/timwu20/gossamer/issues/1536)) ([1a3dea2](1a3dea2)) * **lib/babe:** fix err log ([ChainSafe#1801](https://github.com/timwu20/gossamer/issues/1801)) ([a96f06a](a96f06a)) * **lib/babe:** fix setting first slot of network, fix loading BABE epoch params ([ChainSafe#1640](https://github.com/timwu20/gossamer/issues/1640)) ([5c3dbfe](5c3dbfe)) * **lib/babe:** fix timing for transition between epochs ([ChainSafe#1636](https://github.com/timwu20/gossamer/issues/1636)) ([57027db](57027db)) * **lib/blocktree:** fix blocktree bug ([ChainSafe#2060](https://github.com/timwu20/gossamer/issues/2060)) ([c17b53a](c17b53a)) * **lib/blocktree:** fix potential nil pointer dereference in `HighestCommonAncestor`, core `handleBlocksAsync` ([ChainSafe#1993](https://github.com/timwu20/gossamer/issues/1993)) ([f7f4463](f7f4463)) * **lib/blocktree:** fix setting leaves after blocktree pruning ([ChainSafe#1605](https://github.com/timwu20/gossamer/issues/1605)) ([58c0854](58c0854)) * **lib/blocktree:** removes the inconsistency to choose a deepest leaf ([ChainSafe#2094](https://github.com/timwu20/gossamer/issues/2094)) ([43d68e3](43d68e3)) * **lib/genesis:** Update missing and incorrect fields in genesis file. ([ChainSafe#1681](https://github.com/timwu20/gossamer/issues/1681)) ([8207704](8207704)) * **lib/grandpa:** fix grandpa stall and various bugs ([ChainSafe#1708](https://github.com/timwu20/gossamer/issues/1708)) ([67c93f4](67c93f4)) * **lib/grandpa:** fix grandpa vote message switch ([ChainSafe#2095](https://github.com/timwu20/gossamer/issues/2095)) ([461890c](461890c)) * **lib/grandpa:** fix threshold checking to be strictly greater than 2/3 ([ChainSafe#1891](https://github.com/timwu20/gossamer/issues/1891)) ([66ffe51](66ffe51)) * **lib/grandpa:** use `defaultGrandpaInterval` if not set, fixes error on startup ([ChainSafe#1982](https://github.com/timwu20/gossamer/issues/1982)) ([75627b5](75627b5)) * **lib/runtime/life:** remove import C from life ([ChainSafe#1923](https://github.com/timwu20/gossamer/issues/1923)) ([ed507d2](ed507d2)) * **lib/runtime:** update HOST_API_TEST_RUNTIME_URL to reference specific commit ([ChainSafe#1885](https://github.com/timwu20/gossamer/issues/1885)) ([666ed06](666ed06)) * **log-levels:** do not ignore configuration file log levels ([ChainSafe#2016](https://github.com/timwu20/gossamer/issues/2016)) ([80879b2](80879b2)) * pending bubble hidden after block included ([ChainSafe#1592](https://github.com/timwu20/gossamer/issues/1592)) ([5826322](5826322)) * persist node name ([ChainSafe#1543](https://github.com/timwu20/gossamer/issues/1543)) ([88b88f2](88b88f2)) * **pprof:** only run pprof service if enabled ([ChainSafe#2073](https://github.com/timwu20/gossamer/issues/2073)) ([55669c5](55669c5)) * **release:** Trigger release when pushed to main branch. ([ChainSafe#1566](https://github.com/timwu20/gossamer/issues/1566)) ([d445c97](d445c97)) * **rpc/subscription:** subscribe runtime version notify when version changes ([ChainSafe#1686](https://github.com/timwu20/gossamer/issues/1686)) ([9a76d39](9a76d39)) * Staging CI workflow ([ChainSafe#2034](https://github.com/timwu20/gossamer/issues/2034)) ([84ec792](84ec792)) * **trie:** memory leak fix in `lib/trie` ([ChainSafe#2009](https://github.com/timwu20/gossamer/issues/2009)) ([0ad5eb7](0ad5eb7)) * update deprecated package ([ChainSafe#1603](https://github.com/timwu20/gossamer/issues/1603)) ([f195204](f195204)) * update go-schnorrkel version ([ChainSafe#1557](https://github.com/timwu20/gossamer/issues/1557)) ([b86c7ff](b86c7ff)) * update gssmr genesis to use pallet_babe::SameAuthoritiesForever ([ChainSafe#1696](https://github.com/timwu20/gossamer/issues/1696)) ([fb0a751](fb0a751)) * update HOST_API_TEST_RUNTIME_URL ([ChainSafe#1898](https://github.com/timwu20/gossamer/issues/1898)) ([2ef59a8](2ef59a8)) * **utils:** create a specific folder for database ([ChainSafe#1598](https://github.com/timwu20/gossamer/issues/1598)) ([8c67795](8c67795)) ### Features * add --chain dev option ([ChainSafe#1561](https://github.com/timwu20/gossamer/issues/1561)) ([04a2969](04a2969)) * Add properties and chainId on build-spec command ([ChainSafe#1520](https://github.com/timwu20/gossamer/issues/1520)) ([b18290c](b18290c)) * **cmd/gossamer:** implement --telemetry-url parameter ([ChainSafe#1890](https://github.com/timwu20/gossamer/issues/1890)) ([b202e89](b202e89)), closes [ChainSafe#1502](https://github.com/timwu20/gossamer/issues/1502) * **cmd:** implement offline pruning of state trie ([ChainSafe#1564](https://github.com/timwu20/gossamer/issues/1564)) ([af9c925](af9c925)) * **devnet:** Local Gossamer Devnet ([ChainSafe#2008](https://github.com/timwu20/gossamer/issues/2008)) ([a520001](a520001)) * **dot/network, lib/grandpa:** request justification on receiving NeighbourMessage, verify justification on receipt ([ChainSafe#1529](https://github.com/timwu20/gossamer/issues/1529)) ([e1f9f42](e1f9f42)) * **dot/network:** add propagate return bool to messageHandler func type to determine whether to propagate message or not ([ChainSafe#1555](https://github.com/timwu20/gossamer/issues/1555)) ([0d6f488](0d6f488)) * **dot/network:** implement streamManager to cleanup not recently used streams ([ChainSafe#1611](https://github.com/timwu20/gossamer/issues/1611)) ([ba861bf](ba861bf)) * **dot/peerset:** Implement peer scoring ([ChainSafe#1791](https://github.com/timwu20/gossamer/issues/1791)) ([1c989ad](1c989ad)) * **dot/rpc/modules:** add `system_addReservedPeer` and `system_removeReservedPeer` RPC call ([ChainSafe#1712](https://github.com/timwu20/gossamer/issues/1712)) ([dba5922](dba5922)) * **dot/rpc:** Add `system_localListenAddresses` RPC call ([ChainSafe#1689](https://github.com/timwu20/gossamer/issues/1689)) ([c981d2e](c981d2e)) * **dot/rpc:** Implement `childstate_getKeys` rpc call ([ChainSafe#1800](https://github.com/timwu20/gossamer/issues/1800)) ([9b2f41e](9b2f41e)) * **dot/rpc:** implement sync_state_genSyncSpec RPC call ([ChainSafe#1827](https://github.com/timwu20/gossamer/issues/1827)) ([2186caf](2186caf)) * **dot/state:** implement online pruning of historical state tries ([ChainSafe#1596](https://github.com/timwu20/gossamer/issues/1596)) ([3eb9399](3eb9399)) * **dot/sync:** implement codeSubstitutes ([ChainSafe#1635](https://github.com/timwu20/gossamer/issues/1635)) ([d87aaeb](d87aaeb)) * **dot/telemetry:** Added connection retry ([ChainSafe#1904](https://github.com/timwu20/gossamer/issues/1904)) ([579a791](579a791)) * **dot/telemetry:** Added more telemetry messages in grandpa client ([ChainSafe#2043](https://github.com/timwu20/gossamer/issues/2043)) ([2e57d15](2e57d15)), closes [ChainSafe#1841](https://github.com/timwu20/gossamer/issues/1841) [ChainSafe#1842](https://github.com/timwu20/gossamer/issues/1842) * **dot/telemetry:** implement notify.finalized telemetry interface ([ChainSafe#1877](https://github.com/timwu20/gossamer/issues/1877)) ([de1a60d](de1a60d)) * **dot/telemetry:** implement substrate_number_leaves metrics ([ChainSafe#1926](https://github.com/timwu20/gossamer/issues/1926)) ([69823c0](69823c0)) * **dot/telemetry:** implement telemetry message network_state ([ChainSafe#1618](https://github.com/timwu20/gossamer/issues/1618)) ([a81844e](a81844e)) * **flags:** read log levels from flags ([ChainSafe#1953](https://github.com/timwu20/gossamer/issues/1953)) ([9694e46](9694e46)) * implement ext_default_child_storage_storage_kill_version_2 ([ChainSafe#1799](https://github.com/timwu20/gossamer/issues/1799)) ([c2908ae](c2908ae)) * implement ext_offchain_index_set_version_1 for wasmer runtime ([ChainSafe#1739](https://github.com/timwu20/gossamer/issues/1739)) ([96c30a6](96c30a6)) * **lib/babe:** add check of types.ConfigData.SecondarySlots for disabling secondary verification ([ChainSafe#1910](https://github.com/timwu20/gossamer/issues/1910)) ([cd27ae4](cd27ae4)) * **lib/grandpa:** fully verify justifications using GrandpaState ([ChainSafe#1544](https://github.com/timwu20/gossamer/issues/1544)) ([028d25e](028d25e)) * **lib/grandpa:** Include equivocatory nodes while creating justification ([ChainSafe#1911](https://github.com/timwu20/gossamer/issues/1911)) ([aca86b6](aca86b6)) * **lib/grandpa:** send NeighbourMessage to peers ([ChainSafe#1558](https://github.com/timwu20/gossamer/issues/1558)) ([322ccf9](322ccf9)) * **lib/runtime/wasmer:** implement ext_default_child_storage_storage_kill_version_3 ([ChainSafe#1878](https://github.com/timwu20/gossamer/issues/1878)) ([a719a60](a719a60)) * **lib/runtime/wasmer:** implement ext_offchain_local_storage_version_1 ([ChainSafe#1821](https://github.com/timwu20/gossamer/issues/1821)) ([0f63b17](0f63b17)) * **lib/runtime:** Implement `ext_offchain_http_request_add_header_version_1` host function ([ChainSafe#1994](https://github.com/timwu20/gossamer/issues/1994)) ([0a30b3d](0a30b3d)) * **lib/runtime:** Implement `ext_offchain_http_request_start_version_1` host function ([ChainSafe#1947](https://github.com/timwu20/gossamer/issues/1947)) ([974b1fc](974b1fc)) * **lib/runtime:** Implement `trie_blake2_256_verify_proof` host function ([ChainSafe#1920](https://github.com/timwu20/gossamer/issues/1920)) ([506565d](506565d)) * **lib/trie:** Implement `verify_proof` function ([ChainSafe#1883](https://github.com/timwu20/gossamer/issues/1883)) ([67bb5ef](67bb5ef)) * **lib/trie:** Implement limit for trie.ClearPrefix ([ChainSafe#1905](https://github.com/timwu20/gossamer/issues/1905)) ([becec9e](becec9e)) * **lib/trie:** Parallel hash trie. ([ChainSafe#1657](https://github.com/timwu20/gossamer/issues/1657)) ([22827e7](22827e7)) * **pprof:** Pprof HTTP server service ([ChainSafe#1991](https://github.com/timwu20/gossamer/issues/1991)) ([ce24ea9](ce24ea9)) * **rpc/subscription:** implement state_unsubscribeStorage ([ChainSafe#1574](https://github.com/timwu20/gossamer/issues/1574)) ([7574f10](7574f10)) * **rpc:** Implement `childstate_getChildStorage` RPC call ([ChainSafe#1832](https://github.com/timwu20/gossamer/issues/1832)) ([3d949f2](3d949f2)) * **rpc:** Implement `childstate_getStorageHash` RPC call ([ChainSafe#1805](https://github.com/timwu20/gossamer/issues/1805)) ([e539bd3](e539bd3)) * **rpc:** Implement `childstate_getStorageSize` RPC call ([ChainSafe#1810](https://github.com/timwu20/gossamer/issues/1810)) ([a04deb6](a04deb6)) * **rpc:** Implement `payment_queryInfo` RPC call ([ChainSafe#1826](https://github.com/timwu20/gossamer/issues/1826)) ([7a5deec](7a5deec)) * **rpc:** Implement `state_getReadProof` rpc call ([ChainSafe#1768](https://github.com/timwu20/gossamer/issues/1768)) ([865f80f](865f80f)) * **runtime:** implement custom logging handler that print function name ([ChainSafe#1825](https://github.com/timwu20/gossamer/issues/1825)) ([2b1276d](2b1276d)) * **telemetry:** send telemetry messages when GRANDPA receieves commit or vote messages ([ChainSafe#2015](https://github.com/timwu20/gossamer/issues/2015)) ([7bf40e1](7bf40e1)), closes [ChainSafe#1840](https://github.com/timwu20/gossamer/issues/1840) [ChainSafe#1839](https://github.com/timwu20/gossamer/issues/1839) [ChainSafe#1838](https://github.com/timwu20/gossamer/issues/1838) * **telemetry:** send txpool.import telemetry msg ([ChainSafe#1966](https://github.com/timwu20/gossamer/issues/1966)) ([ffc81bf](ffc81bf)) ### Reverts * Revert "feat(dot/rpc) implement `author_hasSessionKeys` RPC call (ChainSafe#1704)" (ChainSafe#1714) ([65380fd](65380fd)), closes [ChainSafe#1704](https://github.com/timwu20/gossamer/issues/1704) [ChainSafe#1714](https://github.com/timwu20/gossamer/issues/1714)

- Add a pprof HTTP server service to dot node - `internal/httpserver` package - `internal/pprof` package - Configurable server listening address from flags and toml config file - Configurable pprof server on/off from flags and toml config file - Configurable mutex profile fraction from flags and toml config file - Configurable block profile rate from flags and toml config file - Pprof documentation - Remove pprof to file operation and `--cpuprof` and `--memprof` flags

# [0.6.0](v0.5.0...v0.6.0) (2021-12-06) ### Bug Fixes * **ci:** add missing go-version matrix to fix development branch CI ([ChainSafe#2037](https://github.com/timwu20/gossamer/issues/2037)) ([dfc6ce6](dfc6ce6)) * **cmd/cfg:** Use Babe Lead value from toml config ([ChainSafe#2032](https://github.com/timwu20/gossamer/issues/2032)) ([406d083](406d083)) * confirm block import notifier is closed properly ([ChainSafe#1736](https://github.com/timwu20/gossamer/issues/1736)) ([ce4b9d7](ce4b9d7)) * **docs:** improve build-spec usage docs ([ChainSafe#1706](https://github.com/timwu20/gossamer/issues/1706)) ([65b4753](65b4753)) * **dot/core:** Add only extrinsic during chain reorg. ([ChainSafe#1609](https://github.com/timwu20/gossamer/issues/1609)) ([16340ef](16340ef)) * **dot/core:** Batch process transaction message. ([ChainSafe#1780](https://github.com/timwu20/gossamer/issues/1780)) ([3d209a3](3d209a3)) * **dot/core:** check transaction Validity.Propagate field to determine whether to propagate tx ([ChainSafe#1643](https://github.com/timwu20/gossamer/issues/1643)) ([81c8f0a](81c8f0a)) * **dot/core:** Fix handle transaction message test. ([ChainSafe#1607](https://github.com/timwu20/gossamer/issues/1607)) ([dca9ad7](dca9ad7)) * **dot/network, lib/grandpa:** fix handshake decoding and grandpa message handler sigabort ([ChainSafe#1631](https://github.com/timwu20/gossamer/issues/1631)) ([b02525b](b02525b)) * **dot/network, lib/grandpa:** fix node sync, improve devnet finality ([de91262](de91262)) * **dot/network:** add nil checks in connManager ([ChainSafe#2069](https://github.com/timwu20/gossamer/issues/2069)) ([09e07cf](09e07cf)) * **dot/network:** Check for size when decoding leb128. ([ChainSafe#1634](https://github.com/timwu20/gossamer/issues/1634)) ([c20a83c](c20a83c)) * **dot/network:** check if peer supports protocol ([ChainSafe#1617](https://github.com/timwu20/gossamer/issues/1617)) ([fb26dd2](fb26dd2)) * **dot/network:** decrease DHT find peers interval for gssmr nodes ([ChainSafe#1703](https://github.com/timwu20/gossamer/issues/1703)) ([7823c57](7823c57)) * **dot/network:** fix bugs in notifications protocol handlers; add metrics for inbound/outbound streams ([ChainSafe#2010](https://github.com/timwu20/gossamer/issues/2010)) ([88174ca](88174ca)) * **dot/network:** fix dht connection on discovery on devnet ([ChainSafe#2059](https://github.com/timwu20/gossamer/issues/2059)) ([d9008aa](d9008aa)) * **dot/network:** fix discovery between gossamer nodes ([ChainSafe#1594](https://github.com/timwu20/gossamer/issues/1594)) ([ec59e3b](ec59e3b)) * **dot/network:** fix memory allocations with `sizedBufferPool` ([ChainSafe#1963](https://github.com/timwu20/gossamer/issues/1963)) ([ddd96f7](ddd96f7)) * **dot/network:** Fix missing digest in header ([ChainSafe#2092](https://github.com/timwu20/gossamer/issues/2092)) ([6f34b1b](6f34b1b)) * **dot/network:** fix stream manager tests ([ChainSafe#1683](https://github.com/timwu20/gossamer/issues/1683)) ([2b4353c](2b4353c)) * **dot/network:** implement a handshake timeout ([ChainSafe#1615](https://github.com/timwu20/gossamer/issues/1615)) ([4ab856e](4ab856e)) * **dot/network:** Implement time based handle transaction ([ChainSafe#1942](https://github.com/timwu20/gossamer/issues/1942)) ([72455d1](72455d1)) * **dot/network:** move low reputation peer removal from network ConnManager to peer scoring logic (dot/peerstate) ([ChainSafe#2068](https://github.com/timwu20/gossamer/issues/2068)) ([8ea3d9f](8ea3d9f)), closes [ChainSafe#2039](https://github.com/timwu20/gossamer/issues/2039) * **dot/network:** Return on EOF error while reading stream. ([ChainSafe#1733](https://github.com/timwu20/gossamer/issues/1733)) ([a9d32fb](a9d32fb)) * **dot/node:** Start websocket server only with `--ws` flag ([ChainSafe#1671](https://github.com/timwu20/gossamer/issues/1671)) ([fd43774](fd43774)) * **dot/state, lib/babe, lib/trie:** improve syncing between gossamer authority nodes ([ChainSafe#1613](https://github.com/timwu20/gossamer/issues/1613)) ([2e62a3d](2e62a3d)) * **dot/state, lib/grandpa:** update justification and SignedVote handling in database ([ChainSafe#1682](https://github.com/timwu20/gossamer/issues/1682)) ([9dda6b5](9dda6b5)) * **dot/state:** add StorageState Lock/Unlock API for usage by babe and sync ([ChainSafe#1700](https://github.com/timwu20/gossamer/issues/1700)) ([eaf8ba9](eaf8ba9)) * **dot/state:** fix deadlock, fixes bootstrap syncing ([ChainSafe#1959](https://github.com/timwu20/gossamer/issues/1959)) ([13c92ea](13c92ea)) * **dot/state:** track runtime per-block, fix runtime upgrades differing between forks ([ChainSafe#1638](https://github.com/timwu20/gossamer/issues/1638)) ([b31bba5](b31bba5)) * **dot/state:** update `*state.BlockState.AddBlockToBlockTree` to store block in `unfinalisedBlocksMap` ([ChainSafe#2006](https://github.com/timwu20/gossamer/issues/2006)) ([8fac690](8fac690)) * **dot/sync:** add nil header checks ([ChainSafe#2099](https://github.com/timwu20/gossamer/issues/2099)) ([abbefce](abbefce)) * **dot/sync:** fix block request and response logic ([ChainSafe#1907](https://github.com/timwu20/gossamer/issues/1907)) ([fcfbabe](fcfbabe)) * **dot/sync:** fix creating block response, fixes node sync between gossamer nodes ([ChainSafe#1572](https://github.com/timwu20/gossamer/issues/1572)) ([d2197b0](d2197b0)) * **dot/telemetry:** refactor telemetry to reduce CPU usage ([ChainSafe#1597](https://github.com/timwu20/gossamer/issues/1597)) ([3445d84](3445d84)) * **dot/types:** *types.Body to be of type []types.Extrinsic ([ChainSafe#1807](https://github.com/timwu20/gossamer/issues/1807)) ([953d6f4](953d6f4)) * **dot/types:** fix max value for digest ([ChainSafe#1687](https://github.com/timwu20/gossamer/issues/1687)) ([13380af](13380af)) * **dot:** fix `TestNewNode` ([ChainSafe#2070](https://github.com/timwu20/gossamer/issues/2070)) ([87f41ba](87f41ba)) * fix Kusama sync; add storageState lock in core.HandleTransactionMessage ([ChainSafe#1783](https://github.com/timwu20/gossamer/issues/1783)) ([d03f6d1](d03f6d1)) * **lib/babe, lib/runtime/wasmer:** fixes for v0.9.8+ runtime ([ChainSafe#2075](https://github.com/timwu20/gossamer/issues/2075)) ([eb963d3](eb963d3)) * **lib/babe:** add `--babe-lead` flag, update epoch handling logic ([ChainSafe#1895](https://github.com/timwu20/gossamer/issues/1895)) ([8a65c7f](8a65c7f)) * **lib/babe:** add pre-runtime digest before calling initialize_block ([ChainSafe#1581](https://github.com/timwu20/gossamer/issues/1581)) ([f0060ae](f0060ae)) * **lib/babe:** always use 2/3 of slot to produce block, re-add potentially valid txs to queue ([ChainSafe#1679](https://github.com/timwu20/gossamer/issues/1679)) ([b224f10](b224f10)) * **lib/babe:** call AddBlock in BABE synchronously ([ChainSafe#1585](https://github.com/timwu20/gossamer/issues/1585)) ([549dc25](549dc25)) * **lib/babe:** fix err log ([ChainSafe#1801](https://github.com/timwu20/gossamer/issues/1801)) ([0ab395b](0ab395b)) * **lib/babe:** fix setting first slot of network, fix loading BABE epoch params ([ChainSafe#1640](https://github.com/timwu20/gossamer/issues/1640)) ([63f73ca](63f73ca)) * **lib/babe:** fix timing for transition between epochs ([ChainSafe#1636](https://github.com/timwu20/gossamer/issues/1636)) ([e67b1e2](e67b1e2)) * **lib/blocktree:** fix blocktree bug ([ChainSafe#2060](https://github.com/timwu20/gossamer/issues/2060)) ([de8f573](de8f573)) * **lib/blocktree:** fix potential nil pointer dereference in `HighestCommonAncestor`, core `handleBlocksAsync` ([ChainSafe#1993](https://github.com/timwu20/gossamer/issues/1993)) ([7006320](7006320)) * **lib/blocktree:** fix setting leaves after blocktree pruning ([ChainSafe#1605](https://github.com/timwu20/gossamer/issues/1605)) ([a00f881](a00f881)) * **lib/blocktree:** removes the inconsistency to choose a deepest leaf ([ChainSafe#2094](https://github.com/timwu20/gossamer/issues/2094)) ([5210dc0](5210dc0)) * **lib/genesis:** Update missing and incorrect fields in genesis file. ([ChainSafe#1681](https://github.com/timwu20/gossamer/issues/1681)) ([f00a5a2](f00a5a2)) * **lib/grandpa:** fix grandpa stall and various bugs ([ChainSafe#1708](https://github.com/timwu20/gossamer/issues/1708)) ([fe67586](fe67586)) * **lib/grandpa:** fix grandpa vote message switch ([ChainSafe#2095](https://github.com/timwu20/gossamer/issues/2095)) ([8312490](8312490)) * **lib/grandpa:** fix threshold checking to be strictly greater than 2/3 ([ChainSafe#1891](https://github.com/timwu20/gossamer/issues/1891)) ([ed51f97](ed51f97)) * **lib/grandpa:** use `defaultGrandpaInterval` if not set, fixes error on startup ([ChainSafe#1982](https://github.com/timwu20/gossamer/issues/1982)) ([84e2005](84e2005)) * **lib/runtime/life:** remove import C from life ([ChainSafe#1923](https://github.com/timwu20/gossamer/issues/1923)) ([b93152a](b93152a)) * **lib/runtime:** update HOST_API_TEST_RUNTIME_URL to reference specific commit ([ChainSafe#1885](https://github.com/timwu20/gossamer/issues/1885)) ([e6689e8](e6689e8)) * **log-levels:** do not ignore configuration file log levels ([ChainSafe#2016](https://github.com/timwu20/gossamer/issues/2016)) ([44c9b7a](44c9b7a)) * pending bubble hidden after block included ([ChainSafe#1592](https://github.com/timwu20/gossamer/issues/1592)) ([16e7eef](16e7eef)) * **pprof:** only run pprof service if enabled ([ChainSafe#2073](https://github.com/timwu20/gossamer/issues/2073)) ([f45eff4](f45eff4)) * **rpc/subscription:** subscribe runtime version notify when version changes ([ChainSafe#1686](https://github.com/timwu20/gossamer/issues/1686)) ([d6e9ab0](d6e9ab0)) * Staging CI workflow ([ChainSafe#2034](https://github.com/timwu20/gossamer/issues/2034)) ([aaa633c](aaa633c)) * **trie:** memory leak fix in `lib/trie` ([ChainSafe#2009](https://github.com/timwu20/gossamer/issues/2009)) ([71a8a4f](71a8a4f)) * update deprecated package ([ChainSafe#1603](https://github.com/timwu20/gossamer/issues/1603)) ([ee5a5b6](ee5a5b6)) * update gssmr genesis to use pallet_babe::SameAuthoritiesForever ([ChainSafe#1696](https://github.com/timwu20/gossamer/issues/1696)) ([281eef2](281eef2)) * update HOST_API_TEST_RUNTIME_URL ([ChainSafe#1898](https://github.com/timwu20/gossamer/issues/1898)) ([afe350c](afe350c)) * **utils:** create a specific folder for database ([ChainSafe#1598](https://github.com/timwu20/gossamer/issues/1598)) ([4720d27](4720d27)) ### Features * add --chain dev option ([ChainSafe#1561](https://github.com/timwu20/gossamer/issues/1561)) ([dc8ee9d](dc8ee9d)) * **cmd/gossamer:** implement --telemetry-url parameter ([ChainSafe#1890](https://github.com/timwu20/gossamer/issues/1890)) ([154d6e2](154d6e2)), closes [ChainSafe#1502](https://github.com/timwu20/gossamer/issues/1502) * **cmd:** implement offline pruning of state trie ([ChainSafe#1564](https://github.com/timwu20/gossamer/issues/1564)) ([d4d61c7](d4d61c7)) * **devnet:** Local Gossamer Devnet ([ChainSafe#2008](https://github.com/timwu20/gossamer/issues/2008)) ([23ce39f](23ce39f)) * **dot/network:** implement streamManager to cleanup not recently used streams ([ChainSafe#1611](https://github.com/timwu20/gossamer/issues/1611)) ([005932b](005932b)) * **dot/peerset:** Implement peer scoring ([ChainSafe#1791](https://github.com/timwu20/gossamer/issues/1791)) ([05f7e32](05f7e32)) * **dot/rpc/modules:** add `system_addReservedPeer` and `system_removeReservedPeer` RPC call ([ChainSafe#1712](https://github.com/timwu20/gossamer/issues/1712)) ([79c250e](79c250e)) * **dot/rpc:** Add `system_localListenAddresses` RPC call ([ChainSafe#1689](https://github.com/timwu20/gossamer/issues/1689)) ([0d199b5](0d199b5)) * **dot/rpc:** Implement `childstate_getKeys` rpc call ([ChainSafe#1800](https://github.com/timwu20/gossamer/issues/1800)) ([c5181ae](c5181ae)) * **dot/rpc:** implement sync_state_genSyncSpec RPC call ([ChainSafe#1827](https://github.com/timwu20/gossamer/issues/1827)) ([9991a9c](9991a9c)) * **dot/state:** implement online pruning of historical state tries ([ChainSafe#1596](https://github.com/timwu20/gossamer/issues/1596)) ([31700c6](31700c6)) * **dot/sync:** implement codeSubstitutes ([ChainSafe#1635](https://github.com/timwu20/gossamer/issues/1635)) ([1a238b1](1a238b1)) * **dot/telemetry:** Added connection retry ([ChainSafe#1904](https://github.com/timwu20/gossamer/issues/1904)) ([1c0c5c7](1c0c5c7)) * **dot/telemetry:** Added more telemetry messages in grandpa client ([ChainSafe#2043](https://github.com/timwu20/gossamer/issues/2043)) ([7c870a4](7c870a4)), closes [ChainSafe#1841](https://github.com/timwu20/gossamer/issues/1841) [ChainSafe#1842](https://github.com/timwu20/gossamer/issues/1842) * **dot/telemetry:** implement notify.finalized telemetry interface ([ChainSafe#1877](https://github.com/timwu20/gossamer/issues/1877)) ([b4e5ded](b4e5ded)) * **dot/telemetry:** implement substrate_number_leaves metrics ([ChainSafe#1926](https://github.com/timwu20/gossamer/issues/1926)) ([847de6a](847de6a)) * **dot/telemetry:** implement telemetry message network_state ([ChainSafe#1618](https://github.com/timwu20/gossamer/issues/1618)) ([73e99c9](73e99c9)) * **flags:** read log levels from flags ([ChainSafe#1953](https://github.com/timwu20/gossamer/issues/1953)) ([abf9f3a](abf9f3a)) * implement ext_default_child_storage_storage_kill_version_2 ([ChainSafe#1799](https://github.com/timwu20/gossamer/issues/1799)) ([541f580](541f580)) * implement ext_offchain_index_set_version_1 for wasmer runtime ([ChainSafe#1739](https://github.com/timwu20/gossamer/issues/1739)) ([619306d](619306d)) * **lib/babe:** add check of types.ConfigData.SecondarySlots for disabling secondary verification ([ChainSafe#1910](https://github.com/timwu20/gossamer/issues/1910)) ([7bc2e32](7bc2e32)) * **lib/grandpa:** Include equivocatory nodes while creating justification ([ChainSafe#1911](https://github.com/timwu20/gossamer/issues/1911)) ([af7725f](af7725f)) * **lib/runtime/wasmer:** implement ext_default_child_storage_storage_kill_version_3 ([ChainSafe#1878](https://github.com/timwu20/gossamer/issues/1878)) ([454689b](454689b)) * **lib/runtime/wasmer:** implement ext_offchain_local_storage_version_1 ([ChainSafe#1821](https://github.com/timwu20/gossamer/issues/1821)) ([01dbffc](01dbffc)) * **lib/runtime:** Implement `ext_offchain_http_request_add_header_version_1` host function ([ChainSafe#1994](https://github.com/timwu20/gossamer/issues/1994)) ([2e722c7](2e722c7)) * **lib/runtime:** Implement `ext_offchain_http_request_start_version_1` host function ([ChainSafe#1947](https://github.com/timwu20/gossamer/issues/1947)) ([4a8598b](4a8598b)) * **lib/runtime:** Implement `trie_blake2_256_verify_proof` host function ([ChainSafe#1920](https://github.com/timwu20/gossamer/issues/1920)) ([1bd33a6](1bd33a6)) * **lib/trie:** Implement `verify_proof` function ([ChainSafe#1883](https://github.com/timwu20/gossamer/issues/1883)) ([68650b7](68650b7)) * **lib/trie:** Implement limit for trie.ClearPrefix ([ChainSafe#1905](https://github.com/timwu20/gossamer/issues/1905)) ([9756b2c](9756b2c)) * **lib/trie:** Parallel hash trie. ([ChainSafe#1657](https://github.com/timwu20/gossamer/issues/1657)) ([22f0b2b](22f0b2b)) * **pprof:** Pprof HTTP server service ([ChainSafe#1991](https://github.com/timwu20/gossamer/issues/1991)) ([5996229](5996229)) * **rpc/subscription:** implement state_unsubscribeStorage ([ChainSafe#1574](https://github.com/timwu20/gossamer/issues/1574)) ([56f594a](56f594a)) * **rpc:** Implement `childstate_getChildStorage` RPC call ([ChainSafe#1832](https://github.com/timwu20/gossamer/issues/1832)) ([4977d06](4977d06)) * **rpc:** Implement `childstate_getStorageHash` RPC call ([ChainSafe#1805](https://github.com/timwu20/gossamer/issues/1805)) ([0e31754](0e31754)) * **rpc:** Implement `childstate_getStorageSize` RPC call ([ChainSafe#1810](https://github.com/timwu20/gossamer/issues/1810)) ([77f3458](77f3458)) * **rpc:** Implement `payment_queryInfo` RPC call ([ChainSafe#1826](https://github.com/timwu20/gossamer/issues/1826)) ([d6e3583](d6e3583)) * **rpc:** Implement `state_getReadProof` rpc call ([ChainSafe#1768](https://github.com/timwu20/gossamer/issues/1768)) ([00c4951](00c4951)) * **runtime:** implement custom logging handler that print function name ([ChainSafe#1825](https://github.com/timwu20/gossamer/issues/1825)) ([5301ce4](5301ce4)) * **telemetry:** send telemetry messages when GRANDPA receieves commit or vote messages ([ChainSafe#2015](https://github.com/timwu20/gossamer/issues/2015)) ([90d5577](90d5577)), closes [ChainSafe#1840](https://github.com/timwu20/gossamer/issues/1840) [ChainSafe#1839](https://github.com/timwu20/gossamer/issues/1839) [ChainSafe#1838](https://github.com/timwu20/gossamer/issues/1838) * **telemetry:** send txpool.import telemetry msg ([ChainSafe#1966](https://github.com/timwu20/gossamer/issues/1966)) ([9f9c213](9f9c213)) ### Reverts * Revert "feat(dot/rpc) implement `author_hasSessionKeys` RPC call (ChainSafe#1704)" (ChainSafe#1714) ([ed56d29](ed56d29)), closes [ChainSafe#1704](https://github.com/timwu20/gossamer/issues/1704) [ChainSafe#1714](https://github.com/timwu20/gossamer/issues/1714)

I am Quentin Mc Gaw, a software engineer working the Go Polkadot host **Gossamer**. I have been working full time on Gossamer since October 2021, mostly on the state trie and storage. I have also made a [few minor pull requests](https://github.com/w3f/polkadot-spec/pulls?q=is%3Apr+is%3Aclosed+author%3Aqdm12) to the Polkadot specification repository. I am requesting to join the Fellowship at rank 1. ## Main contributions ### Gossamer - Fix memory leaks - Trie encoding buffer pools usage fixed [#2009](ChainSafe/gossamer#2009) - Fix state map of tries memory leak [#2286](ChainSafe/gossamer#2286) - Fix sync benchmark [#2234](ChainSafe/gossamer#2234) - Trie proof fixes ([#2604](ChainSafe/gossamer#2604), [#2661](ChainSafe/gossamer#2661)) - Fix end to end tests orchestration ([#2470](ChainSafe/gossamer#2470), [#2452](ChainSafe/gossamer#2452), [#2385](ChainSafe/gossamer#2385), [#2370](ChainSafe/gossamer#2370)) - State trie statistics ([#2378](ChainSafe/gossamer#2378), [#2310](ChainSafe/gossamer#2310), [#2272](ChainSafe/gossamer#2272)) - State trie fixes and improvements - Only deep copy nodes when mutation is certain [#2352](ChainSafe/gossamer#2352) and [#2223](ChainSafe/gossamer#2223) - Only deep copy necessary fields of a node [#2384](ChainSafe/gossamer#2384) - Use Merkle values for database keys instead of always hash [#2725](ChainSafe/gossamer#2725) - Opportunistic parallel Merkle value commputing [#2081](ChainSafe/gossamer#2081) - Grandpa capped number of tracked messages ([#2490](ChainSafe/gossamer#2490), [#2485](ChainSafe/gossamer#2485)) - Add pprof HTTP service for profiling [#1991](ChainSafe/gossamer#1991) Ongoing work: - State trie lazy loading and caching - State trie v1 support ([#2736](ChainSafe/gossamer#2736), [#2747](ChainSafe/gossamer#2747), [#2687](ChainSafe/gossamer#2687), [#2686](ChainSafe/gossamer#2686), [#2685](ChainSafe/gossamer#2685), [#2673](ChainSafe/gossamer#2673), [#2611](ChainSafe/gossamer#2611), [#2530](ChainSafe/gossamer#2530)) ### Polkadot specification ➡️ [Pull requests from qdm12](https://github.com/w3f/polkadot-spec/pulls?q=is%3Apr+is%3Aclosed+author%3Aqdm12)

qdm12 requested review from arijitAD, EclesioMeloJunior, edwardmack, jimjbrettj, kishansagathiya, noot, omar391 and timwu20 as code owners November 5, 2021 14:27

noot reviewed Nov 5, 2021

View reviewed changes

dot/services.go Outdated Show resolved Hide resolved

noot reviewed Nov 5, 2021

View reviewed changes

internal/httpserver/address.go Show resolved Hide resolved

noot reviewed Nov 5, 2021

View reviewed changes

internal/httpserver/matchers_test.go Show resolved Hide resolved

EclesioMeloJunior approved these changes Nov 8, 2021

View reviewed changes

qdm12 force-pushed the qdm12-mem-pprof-http-server branch from 379a330 to 8985412 Compare November 10, 2021 15:24

noot approved these changes Nov 10, 2021

View reviewed changes

qdm12 force-pushed the qdm12-mem-pprof-http-server branch from 8985412 to dc359bc Compare November 15, 2021 13:32

qdm12 mentioned this pull request Nov 17, 2021

Profile memory usage and describe plan of action to address issues #1936

Closed

qdm12 added 10 commits November 19, 2021 13:58

internal/httpserver package

477b167

internal/pprof package with HTTP server

e631983

Pprof server as a service

ae26f5d

Add pprof service to node

7808ab2

Fix unit test to be working on Ubuntu

83238f6

Add more pprof routes

1a1fda6

Documentation WIP

a9f2758

Fix close of closed channel

d87da97

Add support for /block and /mutex

9624ea5

- Call `runtime.SetBlockProfileRate` - Call `runtime.SetMutexProfileFraction` - Parametrable rates

Configurable pprof server

de96528

- Read from flags - Read from TOML config file - Update toml files with default values

qdm12 added 5 commits November 19, 2021 13:58

Update documentation

292144e

Fix rebase conflicts

7bef31b

Add license tags

b1a8eb6

More documentation on pprof options

8bd126c

Remove pprof to file

ffece83

- because it impacts performance if running all the time - because you can easily download the file from the http server - because it clutters the main code

qdm12 force-pushed the qdm12-mem-pprof-http-server branch from 770c588 to ffece83 Compare November 19, 2021 14:09

qdm12 merged commit ce24ea9 into development Nov 19, 2021

qdm12 deleted the qdm12-mem-pprof-http-server branch November 19, 2021 15:40

qdm12 mentioned this pull request Sep 29, 2022

Member Request polkadot-fellows/seeding#28

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(pprof): Pprof HTTP server service #1991

feat(pprof): Pprof HTTP server service #1991

qdm12 commented Nov 5, 2021 •

edited

Loading

codecov bot commented Nov 5, 2021 •

edited

Loading

EclesioMeloJunior left a comment

qdm12 commented Nov 8, 2021 •

edited

Loading

noot left a comment

qdm12 commented Nov 11, 2021

qdm12 commented Nov 15, 2021 •

edited

Loading

github-actions bot commented Dec 3, 2021

feat(pprof): Pprof HTTP server service #1991

feat(pprof): Pprof HTTP server service #1991

Conversation

qdm12 commented Nov 5, 2021 • edited Loading

Changes

Tests

Issues

Primary Reviewer

codecov bot commented Nov 5, 2021 • edited Loading

Codecov Report

EclesioMeloJunior left a comment

Choose a reason for hiding this comment

qdm12 commented Nov 8, 2021 • edited Loading

noot left a comment

Choose a reason for hiding this comment

qdm12 commented Nov 11, 2021

qdm12 commented Nov 15, 2021 • edited Loading

github-actions bot commented Dec 3, 2021

qdm12 commented Nov 5, 2021 •

edited

Loading

codecov bot commented Nov 5, 2021 •

edited

Loading

qdm12 commented Nov 8, 2021 •

edited

Loading

qdm12 commented Nov 15, 2021 •

edited

Loading