Fix recCeiver issues#153
Merged
Merged
Conversation
anderslindho
force-pushed
the
fix-recceiver-issues
branch
from
0eb1f08 to
7421a18
Compare
jacomago
reviewed
May 6, 2026
Contributor
jacomago
left a comment
jacomago
left a comment
There was a problem hiding this comment.
Could we tests for the non-logging changes?
Contributor
|
Tried to compare against my per ioc lock branch https://github.com/jacomago/recsync/tree/ioc-per-lock . |
Contributor
Author
Yeah, I think the per-IOC lock really is a better solution than this, but this is the more minimal fix that hopefully would settle our production issues @jacomago |
When a session closes, self.C.cancel() was discarding any data commit still queued behind the global DeferredLock. The disconnect transaction then acquired the lock first and marked all channels Inactive in CF, with the channel data never written. Under load with many concurrent IOCs this caused a near-total loss of active-channel registrations. Removing the cancel chains the disconnect transaction after self.C so it executes only once any pending data commit has completed.
Logging the full CFConfig repr at INFO level exposed the CF password in plaintext to anyone with log access. Override __repr__ to replace cf_password with '***' so startup logs remain informative without leaking credentials.
The previous code checked `iocid in self.iocs` before decrementing channelcount but then accessed self.iocs[iocid] unconditionally on the next lines, raising KeyError when an iocid was absent — possible when disconnect transactions race with initial transactions under load. Early-return when iocid is missing; consolidate the channelcount == 0 and < 0 branches so they share a single iocs lookup.
trlimit defaulted to 0, permanently disabling the count-based flush branch in flushSafely. Fast IOCs transmit all records in milliseconds so the timer branch never fires, causing a single giant transaction per IOC — particularly bad for large IOCs that cause Humongous GC on the CF JVM. Set the default to 5000 records so fast large uploads are split automatically. Add DEBUG logging when either limit triggers so the behaviour is observable in practice.
The warning serialised the entire ioc dict — several kilobytes per line under production load. It fired on every reconnect as a direct side-effect of Issue 1, flooding the log aggregation pipeline. Log only the count of known IOCs, which is enough to distinguish a race condition (non-zero) from a completely cold startup (zero).
Silent fallbacks made it hard to find misconfigured IOCs: port numbers appearing as iocName and the recceiver service account appearing as owner were invisible unless you already knew what to look for. Log at DEBUG when either fallback is used so misconfigured IOCs are surfaced without generating noise on every commit.
anderslindho
force-pushed
the
fix-recceiver-issues
branch
from
7421a18 to
4db0d9d
Compare
|
jacomago
approved these changes
May 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See individual commits