Out-of-touch watchers #182
Comments
4 tasks
tapaswenipathak
added a commit
that referenced
this issue
Jun 12, 2018
…queue threshold, info to doc (#211) * Closes #208, #207, #206, #182, #149, #72, #15 (cherry picked from commit 8de328df79ccf52b8d612c625891555808c2fa0e) * Add minSize as option * update jest tests * Change MinSize to 0 * update jest * identation and minSize to 0 * Add deadletterThreshold info in Worker-retry-cycle
jakepruitt
pushed a commit
that referenced
this issue
Jun 16, 2018
…queue threshold, info to doc (#211) * Closes #208, #207, #206, #182, #149, #72, #15 (cherry picked from commit 8de328df79ccf52b8d612c625891555808c2fa0e) * Add minSize as option * update jest tests * Change MinSize to 0 * update jest * identation and minSize to 0 * Add deadletterThreshold info in Worker-retry-cycle
jakepruitt
pushed a commit
that referenced
this issue
Jun 16, 2018
* ♻ that container * Add logging (#185) * adds logging of watcher-level errors, worker receives, and completion status * prefixed logs from child processes * fixes logger factory to accept a message * --> false for legibility * move binary split to dependency * package lock changes * Scale down threshold (#187) * change scale-down MetricIntervalLowerBound to MetricIntervalUpperBound * exit main loop after workers finish * resolve() after all workers return instead of exiting * fix tests and mocks * cleanup * logs * another log * use logger and process.stdout for logs * more logs * edit logs * Add alarms to "♻️ that container" PR (#198) * Add alarms and alarm docs * Add failedPlacementAlarmPeriods * Add CloudWatch Alarms snapshots * Update template jest snapshots * Add CloudWatch Alarms snapshots * Add failedworker and failedworkerplacement metric * Typo r/LogGroup/Logs * Change metric name * Metric Filter of worker errors to "[failure]" * Have current published version instead of undefined * Jake's Review * uh update-jest * Update alarms.md * Add template validation tests (#215) * Add travis user * Ensure this fails * Add validation for notificationEmail or notificationTopic * Add minSize and maxSize of service scaleup and scaledown, deadletter queue threshold, info to doc (#211) * Closes #208, #207, #206, #182, #149, #72, #15 (cherry picked from commit 8de328df79ccf52b8d612c625891555808c2fa0e) * Add minSize as option * update jest tests * Change MinSize to 0 * update jest * identation and minSize to 0 * Add deadletterThreshold info in Worker-retry-cycle * Update tests with maxSize property * remove superfluous logging * add fresh mode as a watchbot option * if else * freshMode * console log * typeof * true * concise * add fresh * fix tests * fix binary test * update snapshots * Allow users to write to any volume (#200) * Restrict writes to volumes and clean them after every job * Try out the `ReadOnlyRootFilesystem` option * Capitalization * Add watchbot-log * use strict * No need to chmod now
|
Going to close this for now, since watchbot 4 controls concurrency through an ECS service rather than through an internal state machine. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I've heard of this happening on multiple watchbot stacks, but saw it first-hand today when I found a watcher logging messages of
{"max":"1","concurrency":1,"messages":1}even though there was no worker task running and all of the queues (WatchbotQueue, WatchbotTaskEventQueue, WatchbotDeadLetterQueue) were completely empty. Something must have slipped in the watcher.cc/ @rclark @dnomadb @jacquestardie
The text was updated successfully, but these errors were encountered: