Add configuration for batch size/polling interval #364
Conversation
Charles-Gagnon
requested review from
MaddyDev,
chlafreniere and
lucyzhang929
as code owners
| { | ||
| public const int BatchSize = 10; | ||
| public const int PollingIntervalInSeconds = 5; | ||
| public const int MaxAttemptCount = 5; |
There was a problem hiding this comment.
Can you make MaxAttemptCount private too?
| private async Task RunChangeConsumptionLoopAsync() | ||
| { | ||
| this._logger.LogDebugWithThreadId("Starting change consumption loop."); | ||
| this._logger.LogInformationWithThreadId($"Starting change consumption loop. BatchSize: {this._batchSize} PollingIntervalMs: {this._pollingIntervalInMs}"); |
There was a problem hiding this comment.
I think such information should be logged at debug level as user won't be interested in general in internal workings of the extension. We can log the evaluated batch size and polling interval as separate message.
There was a problem hiding this comment.
I'm still a little unsure about exactly what should be info and what should be debug, but does this sound reasonable?
info - information useful to the user to see during normal execution
debug - detailed information about the inner workings
The reason I changed this to info was because I was thinking that a user would find it useful to know the batch size and polling interval - especially for the default values which they may not know offhand.
There was a problem hiding this comment.
I'm going to leave this as info for now for the reasons above, and we can come back and do a look over logging in the entire extension pre-GA to make sure everything lines up how we want (I'll make an issue to track)
There was a problem hiding this comment.
I already mentioned that we should log batch size and polling interval but not the rest of the message.
Users normally use the logs to check the history of their own function invocations. The extension should in general log all the information that might be useful for the user during host/listener startup only. Logging "Starting change consumption loop" even if for one-time will only confuse users and will prompt them to check what the message means for them.
There was a problem hiding this comment.
Oh sorry, I misread your original comment. What would you suggest the info message with the batch size/polling interval would be then?
| public const int PollingIntervalInSeconds = 5; | ||
| public const int MaxAttemptCount = 5; | ||
|
|
||
| private const string CONFIG_KEY_SQL_TRIGGER_BATCHSIZE = "Sql_Trigger_BatchSize"; |
There was a problem hiding this comment.
Should we add these to the SqlTriggerConstants file?
| // Considering the polling interval of 5 seconds and batch-size of 20, it should take around 10 seconds to | ||
| // process 40 insert operations. | ||
| this.InsertProducts(1, 40); | ||
| await Task.Delay(TimeSpan.FromSeconds(12)); |
There was a problem hiding this comment.
I plan on coming back and refactoring this logic to not be a hardcoded wait since that's more unstable and less reliable. Will be done separately though since I want to update all the places that are using this kind of logic at once though.
There was a problem hiding this comment.
If you are planning on updating the wait-logic, should I keep the changes in my PR ready to be committed? I have been busy with Hackathon so haven't picked up any work on SQL extension this week.
* Don't publish TSA results on PRs (#327) * Don't publish TSA results on PRs * fix * Add more debug logging (#328) * add debug logging * pr comments + fix test * log queries * Add benchmark project for performance tests (#294) * add benchmark project * fix logger * fix logger * copy functions to benchmark output folder * cleanup * move setup methods to base * add output binding perf * clean up * add README * fix readme * add link to integration tests * typo * cleanup + comments * address pr comments * fix readme * fix integration test base * pr comments * Fix localization/serialization issues (#329) * Fix localization issues * split tests & cleanup * Replace file copy with itemgroup * Add policheck and var for uploading to TSA (#332) * Add policheck * Upload policheck to TSA codebase * Try globbing files * Scan all * Only upload when var set * post analysis * Only on windows * break on warn * testing * re-enable and break on error * Update notification alias * Fix local build errors (#334) * Fix local build errors * Update comments * Fix log error not printing out primary key name (#335) * Don't run Roslyn analyzers on PRs (#336) * Don't run Roslyn analyzers on PRs * fix * Do not process rows if lease acquisition fails (#339) * Fix README link (#342) * Escape special characters in Readme file (#346) * Wait for error inside `StartFunctionsHostAndWaitForError` method (#349) * Add test for multiple triggered functions (#343) * Add retry when attempting to release leases (#340) * Add retry when attempting to release leases * Address comments * Address more comments * Change loop structure according to comment * sql version telemetry data from input bindings (#337) * connection info telemetry data frm input bindings * capture convert event after collecting connection * remove event * first collect telemetry * fix tests * make convert type optional * update xml * add telemetry service * clean up * remove ITelemetryService * undo changes * merge conflict * missed merge conflict * extra line * add null check * capture telemetry after opening connection * correct connection * move telemtery after connection.OpenAsync * open connection on initialize * add xml * address comments * refactor test * add xml comment * update xml * add check back * add connection state check * remove explicit checks * add isDisposed * add comments * revert isDisposed changes * add comment with github issue link * Refactor locking logic in class `SqlTableChangeMonitor` (#357) * cherry-pick c5a746f * Merge commit '76575f29e964f3487684b84e279444279296f5c2' into chgagnon/mergeFromMain1 # Conflicts: # src/SqlAsyncCollector.cs # test/GlobalSuppressions.cs * Add comments to explain LastSyncVersion selection (#358) * Fix * Move test scripts * Constructor cleanup (#362) * Initialize vars at declaration * Remove extra ; * More cleanup * Remove configuration * Add configuration for batch size/polling interval (#364) * Add configuration for batch size/polling interval * PR comments Co-authored-by: Lucy Zhang <luczhan@microsoft.com> Co-authored-by: Jatin Sanghvi <20547963+JatinSanghvi@users.noreply.github.com> Co-authored-by: AmeyaRele <35621237+AmeyaRele@users.noreply.github.com> Co-authored-by: Maddy <12754347+MaddyDev@users.noreply.github.com>
* Don't publish TSA results on PRs (#327) * Don't publish TSA results on PRs * fix * Add more debug logging (#328) * add debug logging * pr comments + fix test * log queries * Add benchmark project for performance tests (#294) * add benchmark project * fix logger * fix logger * copy functions to benchmark output folder * cleanup * move setup methods to base * add output binding perf * clean up * add README * fix readme * add link to integration tests * typo * cleanup + comments * address pr comments * fix readme * fix integration test base * pr comments * Fix localization/serialization issues (#329) * Fix localization issues * split tests & cleanup * Replace file copy with itemgroup * Add policheck and var for uploading to TSA (#332) * Add policheck * Upload policheck to TSA codebase * Try globbing files * Scan all * Only upload when var set * post analysis * Only on windows * break on warn * testing * re-enable and break on error * Update notification alias * Fix local build errors (#334) * Fix local build errors * Update comments * Fix log error not printing out primary key name (#335) * Don't run Roslyn analyzers on PRs (#336) * Don't run Roslyn analyzers on PRs * fix * Do not process rows if lease acquisition fails (#339) * Fix README link (#342) * Escape special characters in Readme file (#346) * Wait for error inside `StartFunctionsHostAndWaitForError` method (#349) * Add test for multiple triggered functions (#343) * Add retry when attempting to release leases (#340) * Add retry when attempting to release leases * Address comments * Address more comments * Change loop structure according to comment * sql version telemetry data from input bindings (#337) * connection info telemetry data frm input bindings * capture convert event after collecting connection * remove event * first collect telemetry * fix tests * make convert type optional * update xml * add telemetry service * clean up * remove ITelemetryService * undo changes * merge conflict * missed merge conflict * extra line * add null check * capture telemetry after opening connection * correct connection * move telemtery after connection.OpenAsync * open connection on initialize * add xml * address comments * refactor test * add xml comment * update xml * add check back * add connection state check * remove explicit checks * add isDisposed * add comments * revert isDisposed changes * add comment with github issue link * Refactor locking logic in class `SqlTableChangeMonitor` (#357) * cherry-pick 6f23a5a * Merge commit 'd594efe88f0f7d32e4dcd31faad22885291d4902' into chgagnon/mergeFromMain1 # Conflicts: # src/SqlAsyncCollector.cs # test/GlobalSuppressions.cs * Add comments to explain LastSyncVersion selection (#358) * Fix * Move test scripts * Constructor cleanup (#362) * Initialize vars at declaration * Remove extra ; * More cleanup * Remove configuration * Add configuration for batch size/polling interval (#364) * Add configuration for batch size/polling interval * PR comments Co-authored-by: Lucy Zhang <luczhan@microsoft.com> Co-authored-by: Jatin Sanghvi <20547963+JatinSanghvi@users.noreply.github.com> Co-authored-by: AmeyaRele <35621237+AmeyaRele@users.noreply.github.com> Co-authored-by: Maddy <12754347+MaddyDev@users.noreply.github.com>
* Add configuration for batch size/polling interval * PR comments
Closes #330
Adds two configuration values that can be used to control the batch size/polling interval.
I considered adding them as parameters to the trigger attribute as well but decided that was unnecessary overhead - it's simple enough to use an app setting for them.
I've created #330 to follow up with changing the defaults, that will require additional testing so wanted to get this in for now and then we can take our time finding new defaults without worrying about people being blocked.