-
Notifications
You must be signed in to change notification settings - Fork 18
Resolving Background Job Alerts
This page provides instruction to address problems reported by background jobs in the #appeals-job-alerts Slack channel.
Also note that there are some jobs that report to the sprint team's Slack channel, (such as StuckVirtualHearingsChecker
to #appeals-tango and ReviewsWithDuplicateEpErrorChecker
in #appeals-foxtrot).
Examine each appeal's task tree and assess if the appeal would be stuck, i.e., there are no active tasks. Historically, these problems result from Bat Team support tickets, so find the ticket and gather some understanding on what the intended state of the appeal should be. If unsure, ask for help -- don't assume.
If the root cause is from Caseflow code, let the respective sprint team know ASAP. There may be some edge case that was not considered for a recent PR.
appeals=AppealsWithNoTasksOrAllTasksOnHoldQuery.new.call
appeals.count # this should match with the number reported in the Slack message
appeals[0].uuid # Search for a URL with this UUID in #appeals-batteam; the associate Bat Team ticket provides useful context
appeals.first.appellant # Determine if this is due to an unrecognized appellant. If so, see note below.
=> #<OtherClaimant id: 12345, ...
# Examine the task tree and determine what needs to be fixed based on the intended state of the appeal
appeals[0].treee
Note: Currently for appeals with an unrecognized appellant, we prevent the creation of dispatch tasks so the Board does not send out a letter -- Slack. Foxtrot plans to implement workflows to unblock these appeals. Do not try to fix these appeals.
Otherwise, one possible solution is to make the on_hold
task to be assigned
: Task.find(task_id).assigned!
Another solution may be to cancel the task. Prior fixes:
- https://dsva.slack.com/archives/CN3FQR4A1/p1624876324034000
- https://dsva.slack.com/archives/CN3FQR4A1/p1617099918013700
Here are notes from a past meeting about an Echo PI project related to this alert.
Related links:
Ask Teams for help, for example see this Slack thread.
Check if the user is listed in Reassign Tasks for Inactive Accounts Part 3 #16275 and remind Echo to address the inactive user. If the user is not listed, let Echo know.
Instructions for distributed appeals (i.e., appeals where the DistributionTask
is completed
) are provided in https://github.com/department-of-veterans-affairs/caseflow/issues/16275#issuecomment-847141197.
If the tasks are related to hearings (which are usually created before an appeal is distributed), let Tango know.
This Metabase table lists tasks assigned to inactive users.
Prior investigation. Also see Clean up open tasks with closed parents #13438.
tasks=Task.open.where.not(closed_at: nil)
# Review them one at a time
tasks[0].tasks
...
# Get some stats regarding the parent tasks
cp = Task.open.joins(:parent).includes(:parent).where.not(parents_tasks: { closed_at: nil })
cp.group(:type).count
# Identify patterns (see "Prior investigation" link above) and fix them
# Document patterns on this wiki page
# Only if you're confident, fix all of them at once:
tasks_to_fix.update_all(closed_at: nil)
- Home
- Acronyms and Glossary
- Caseflow products
- Caseflow Intake
- Caseflow Queue
- Caseflow Reader
- Caseflow eFolder
- Caseflow Hearings
- Caseflow Certification
- Caseflow APIs
- Appeal Status API
- Caseflow Dispatch
-
CSUM Roles
- System Admin
- VHA Team Management
- Active Record Queries Resource
- External Integrations
- Caseflow Demo
- Caseflow ProdTest
- Background
- Stuck Jobs
- VA Notify
- Caseflow-Team
- Frontend Best Practices
- Accessibility
- How-To
- Debugging Tips
- Adding a Feature Flag with FeatureToggle
- Editing AMA issues
- Editing a decision review
- Fixing task trees
- Investigating and diagnosing issues
- Data and Metric Request Workflow
- Exporting and Importing Appeals
- Explain page for Appeals
- Record associations and Foreign Keys
- Upgrading Ruby
- Stuck Appeals
- Testing Action Mailer Messages Locally
- Re-running Seed Files
- Rake Generator for Legacy Appeals
- Manually running Scheduled Jobs
- System Admin UI
- Caseflow Makefile
- Upgrading Postgresql from v11.7 to v14.8 Locally
- VACOLS VM Trigger Fix M1
- Using SlackService to Send a Job Alert
- Technical Talks