Improve re-usability of ringpop membership & PerNamespaceWorker#9321
Improve re-usability of ringpop membership & PerNamespaceWorker#9321
Conversation
8c72508 to
33567ce
Compare
d708085 to
c6be5b5
Compare
| } | ||
|
|
||
| func (wm *perNamespaceWorkerManager) Start( | ||
| func (wm *PerNamespaceWorkerManager) Start( |
There was a problem hiding this comment.
you should change the key in the AddListener call to something unique in case these end up in the same process. I like to use fmt.Sprintf("%p", wm)
There was a problem hiding this comment.
ah gotcha, made the change
c6be5b5 to
835a49c
Compare
This updates the mapping from service name to ringpop role to using a map instead of an if-else chain. The map enables distributed configurability instead of depending on defining all scenarios in this single place.
f2ce416 to
5759690
Compare
| var ( | ||
| serviceNameToServiceTypeEnumMap = map[primitives.ServiceName]persistence.ServiceType{} | ||
| ) | ||
|
|
||
| func RegisterServiceNameToServiceTypeEnum(serviceName primitives.ServiceName, serviceType persistence.ServiceType) { | ||
| serviceNameToServiceTypeEnumMap[serviceName] = serviceType | ||
| } | ||
|
|
||
| func init() { | ||
| RegisterServiceNameToServiceTypeEnum(primitives.AllServices, persistence.All) | ||
| RegisterServiceNameToServiceTypeEnum(primitives.FrontendService, persistence.Frontend) | ||
| RegisterServiceNameToServiceTypeEnum(primitives.InternalFrontendService, persistence.InternalFrontend) | ||
| RegisterServiceNameToServiceTypeEnum(primitives.HistoryService, persistence.History) | ||
| RegisterServiceNameToServiceTypeEnum(primitives.MatchingService, persistence.Matching) | ||
| RegisterServiceNameToServiceTypeEnum(primitives.WorkerService, persistence.Worker) | ||
| } |
There was a problem hiding this comment.
| var ( | |
| serviceNameToServiceTypeEnumMap = map[primitives.ServiceName]persistence.ServiceType{} | |
| ) | |
| func RegisterServiceNameToServiceTypeEnum(serviceName primitives.ServiceName, serviceType persistence.ServiceType) { | |
| serviceNameToServiceTypeEnumMap[serviceName] = serviceType | |
| } | |
| func init() { | |
| RegisterServiceNameToServiceTypeEnum(primitives.AllServices, persistence.All) | |
| RegisterServiceNameToServiceTypeEnum(primitives.FrontendService, persistence.Frontend) | |
| RegisterServiceNameToServiceTypeEnum(primitives.InternalFrontendService, persistence.InternalFrontend) | |
| RegisterServiceNameToServiceTypeEnum(primitives.HistoryService, persistence.History) | |
| RegisterServiceNameToServiceTypeEnum(primitives.MatchingService, persistence.Matching) | |
| RegisterServiceNameToServiceTypeEnum(primitives.WorkerService, persistence.Worker) | |
| } | |
| var serviceNameToServiceTypeEnumMap = map[primitives.ServiceName]persistence.ServiceType{ | |
| primitives.AllServices: persistence.All, | |
| primitives.FrontendService: persistence.Frontend, | |
| primitives.InternalFrontendService: persistence.InternalFrontend, | |
| primitives.HistoryService: persistence.History, | |
| primitives.MatchingService: persistence.Matching, | |
| primitives.WorkerService: persistence.Worker, | |
| } |
There was a problem hiding this comment.
there still needs to be a RegisterServiceNameToServiceTypeEnum, that's the whole point of this change
| RegisterServiceNameToServiceTypeEnum(primitives.WorkerService, persistence.Worker) | ||
| } | ||
|
|
||
| func serviceNameToServiceTypeEnum(name primitives.ServiceName) (persistence.ServiceType, error) { |
There was a problem hiding this comment.
Seems like a redundant helper but 🤷
There was a problem hiding this comment.
wanted to keep the changes contained - can remove it down the line
dnr
left a comment
There was a problem hiding this comment.
couple nitpicks but no blockers, so approving now
| var ( | ||
| serviceNameToServiceTypeEnumMap = map[primitives.ServiceName]persistence.ServiceType{} | ||
| ) | ||
|
|
||
| func RegisterServiceNameToServiceTypeEnum(serviceName primitives.ServiceName, serviceType persistence.ServiceType) { | ||
| serviceNameToServiceTypeEnumMap[serviceName] = serviceType | ||
| } | ||
|
|
||
| func init() { | ||
| RegisterServiceNameToServiceTypeEnum(primitives.AllServices, persistence.All) | ||
| RegisterServiceNameToServiceTypeEnum(primitives.FrontendService, persistence.Frontend) | ||
| RegisterServiceNameToServiceTypeEnum(primitives.InternalFrontendService, persistence.InternalFrontend) | ||
| RegisterServiceNameToServiceTypeEnum(primitives.HistoryService, persistence.History) | ||
| RegisterServiceNameToServiceTypeEnum(primitives.MatchingService, persistence.Matching) | ||
| RegisterServiceNameToServiceTypeEnum(primitives.WorkerService, persistence.Worker) | ||
| } |
There was a problem hiding this comment.
there still needs to be a RegisterServiceNameToServiceTypeEnum, that's the whole point of this change
|
|
||
| func PerNamespaceWorkerManagerProvider(params perNamespaceWorkerManagerInitParams) *PerNamespaceWorkerManager { | ||
| return NewPerNamespaceWorkerManager( | ||
| params.Logger, |
There was a problem hiding this comment.
could just pass params as a whole, so there's fewer places to change when adding dependencies? I guess this separates the fx stuff more cleanly, so either way
There was a problem hiding this comment.
yeah the goal was separating the FX a tiny bit more so it is not hard assumed which fields come FX and which come from somewhere else
Making the PerNamespaceWorker helper from the worker service exported so it can be re-used by other components.
5759690 to
2cebb48
Compare
* origin/main: CHASM: improve support for implementing Terminate method (#9351) Add testhooks package documentation (#9373) Improve re-usability of ringpop membership & PerNamespaceWorker (#9321) Fairness counter: fix heap bug in map counter (#9370) Avoid finalGC when ack level is zero (#9371) Fairness counter: persist top K keys (#9188) Flake Fix: In Reactivation Cache tests, wait for appropriate delays when confirming expected drainage status (#9352) Include transient and speculative WFT events in GetWorkflowExecutionHistoryResponse (#9325) Fix flaky test TestTransitionDuringTransientTask (#9356) Add per-workflow scheduler for history task processing (#9141) Populate currentAttemptScheduledTime on PollActivityTaskQueueResponse for standalone activities (#9333) Standalone activity heartbeating bug fix (#9354) Revert "Last part of making Nexus work OOTB" (#9343) Convert flake report from Python to Go (#9334) Do not enforce payload limits for system nexus endpoint (#9344)
…oralio#9321) ## What changed? This makes the ServiceName to ServiceType mapping in the Ringpop membership implementation more extensible, and exports the worker-service's PerNamespaceWorker. ## Why? As the code base grows, it becomes useful to enable building system services in seperate repositories (just to manage growth). This approach unblocks re-use of membership and PerNamespaceWorker capabilities that are more generically useful than solely inside the server. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s)
Merges temporalio/temporal main branch up to df2e384. Key upstream changes: - Customizable serialization (temporalio#8426): EncodingTypeFromEnv(), EncodingType() on Encoder - Per-workflow scheduler for history task processing (temporalio#9141) - Ringpop membership & PerNamespaceWorker reusability (temporalio#9321) - Transient/speculative WFT events in history response (temporalio#9325) - Per-check diagnostics in DeepHealthCheck API (temporalio#9350) - Fairness counter heap bug fix (temporalio#9370) - System nexus endpoint (temporalio#9002) - Mixed brain non-blocking (temporalio#9406) - interface{} → any across persistence layer - Various CI, test, and chasm improvements DSQL fork code preserved: - TxRetryPolicy/TxRetryMetrics/TxRetryPolicyProvider (common.go) - ExecutionStoreCreator interface and factory wrapping (factory.go) - OCC-aware lockShard bypass (shard.go) - PoolSizeHint ephemeral pool sizing (version_checker.go) - DSQL-safe InitializeSystemNamespaces (metadata_manager.go) - Snowflake ID generator (idgenerator.go) - RegisterPluginAlias (store.go) - Full DSQL plugin (sqlplugin/dsql/)
…oralio#9321) ## What changed? This makes the ServiceName to ServiceType mapping in the Ringpop membership implementation more extensible, and exports the worker-service's PerNamespaceWorker. ## Why? As the code base grows, it becomes useful to enable building system services in seperate repositories (just to manage growth). This approach unblocks re-use of membership and PerNamespaceWorker capabilities that are more generically useful than solely inside the server. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s)
What changed?
This makes the ServiceName to ServiceType mapping in the Ringpop membership implementation more extensible, and exports the worker-service's PerNamespaceWorker.
Why?
As the code base grows, it becomes useful to enable building system services in seperate repositories (just to manage growth).
This approach unblocks re-use of membership and PerNamespaceWorker capabilities that are more generically useful than solely inside the server.
How did you test it?