Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delayed Job Errors #1621

Closed
jessleenyc opened this issue Jan 22, 2019 · 7 comments

Comments

Projects
None yet
3 participants
@jessleenyc
Copy link
Collaborator

commented Jan 22, 2019

Describe the bug

We're seeing this type of error pop up while executing delayed jobs.

Delayed::PerformableMethod (id=#####) FAILED permanently with Delayed::DeserializationError: ActiveRecord::RecordNotFound

Can likely be recreated by deleting an article with lots of comments, anything that triggers a bunch of callbacks.

@maestromac

This comment has been minimized.

Copy link
Collaborator

commented Feb 8, 2019

There are three main DeserializationError I've seen.

  • Article (resolved with #1699)
  • Reaction
  • Follow
@lightalloy

This comment has been minimized.

Copy link
Collaborator

commented Feb 8, 2019

Thanks, Mac. I'll return to this ticket soon.

@lightalloy

This comment has been minimized.

Copy link
Collaborator

commented Feb 19, 2019

I have found 2 situations when these errors may occur:

  • follow created and immediately destroyed
  • reaction created and immediately destroyed
    These operations can be easily done through the interface. So the delayed job remains after the record is destroyed and fails with an error later.
    I'm currently working on fixing these issues.

If you know other situations when the errors may occur, please share.

@lightalloy

This comment has been minimized.

Copy link
Collaborator

commented Feb 22, 2019

I decided to make separate pull requests for the Follows and Reactions not to make a pr huge. Here's the first pr - #1835

benhalpern added a commit that referenced this issue Feb 24, 2019

Fix delayed job errors related to follows #1621 (#1835)
* Add delayed_job_web to dev environment for debugging

* Add specs for the current Follow callbacks implementation

* Move follower touching to ActiveJob

* Spec for the touch followers job

* Move Follow#create_chat_channel to ActiveJob and make the job safe

* Add ActiveJob to send email notifications about follows

* Enqueue SendEmailNotificationJob after the follow is created

* Specs to wnsure jobs are enqueued on Follow creation

* Make CreateChatChannelJob queue name more specific

benhalpern added a commit that referenced this issue Feb 24, 2019

Bust cache on article destroy instead of resave articles #1621 (#1801)
* Bust cache on article destroy instead of resave articles

* Change articles delayed job queue

benhalpern added a commit that referenced this issue Feb 28, 2019

Fix delayed job errors related to reactions #1621 (#1883)
* Add delayed_job_web to dev environment for debugging

* Add specs for the current Follow callbacks implementation

* Move follower touching to ActiveJob

* Spec for the touch followers job

* Move Follow#create_chat_channel to ActiveJob and make the job safe

* Add ActiveJob to send email notifications about follows

* Enqueue SendEmailNotificationJob after the follow is created

* Specs to wnsure jobs are enqueued on Follow creation

* Make CreateChatChannelJob queue name more specific

* Reaction specs for Reaction after_save callbacks logic

* Touch user job

* Call Users::TouchJob on reaction create

* Move updating reactable to a separate job

* Move busting reactable cache to a separate job

* Bust homepage cache after reaction save in a separate job

* Spec for enqueueing Users::TouchJob on reaction create

* Refactor Reactions::UpdateReactable job

* Fix observer spec

Glennmen added a commit to Glennmen/dev.to that referenced this issue Feb 28, 2019

Merge branch 'master' into lint/fix-linting-api-views
* master:
  Fix delayed job errors related to reactions thepracticaldev#1621 (thepracticaldev#1883)
  Change OP to AUTHOR (thepracticaldev#1919)
  add username to author info (thepracticaldev#1912)
  Add test for validation badge image (thepracticaldev#1909) [ci skip]
  Adjust article ranking algorithm to use crossposted_at where available (thepracticaldev#1917)
  Remove unused Follow methods (thepracticaldev#1915) [ci skip]
  Fix linting issues in app/views/dashboards (thepracticaldev#1854) [ci skip]
  Add OP badge on comments (thepracticaldev#461)
  Fix settings/mentorship SyntaxError (thepracticaldev#1913)
  Fix linting issues in app/views/users (thepracticaldev#1908)
  Fix linting issues in app/views/stripe_subscriptions (thepracticaldev#1905) [ci skip]
  Fix linting issues in app/views/podcast_episodes (thepracticaldev#1898) [ci skip]
  Fix linting issues in app/views/moderations (thepracticaldev#1900) [ci skip]
  Fix linting issues in app/views/reading_list_items (thepracticaldev#1902) [ci skip]
  Fix linting issues for app/views/tags (thepracticaldev#1906) [ci skip]
  Fix linting issues in app/views/pages (thepracticaldev#1897)
  Fix linting issues in app/views/social_previews (thepracticaldev#1903) [ci skip]
  Fix linting issues for app/views/stories (thepracticaldev#1904) [ci skip]
  Fix linting issues for app/views/html_variants (thepracticaldev#1872) [ci skip]
  Add missing argument for reaction notification (thepracticaldev#1901)

benhalpern added a commit that referenced this issue Feb 28, 2019

Remove old methods from reactions #1621 (#1924)
* Remove old methods from reactions

* Return the needed Reaction methods
@lightalloy

This comment has been minimized.

Copy link
Collaborator

commented Mar 5, 2019

@jessleenyc @maestromac Since delayed job web interface has been enabled for admins, could you provide me more information on the errors that occur?
It could also be beneficial at this point to keep failed jobs in the database, at least temporarily (I would suggest using an env variable, like

Delayed::Worker.destroy_failed_jobs = ENV.fetch("DESTROY_FAILED_DELAYED_JOBS", true)`

But if you have information on the failed jobs without adding this setting, I would like to examine it first.

@lightalloy

This comment has been minimized.

Copy link
Collaborator

commented Apr 11, 2019

As I see, most of the described Delayed::DeserializationError are solved now. However, they could still appear in some cases, which are not so common.
I'm currently researching on and fixing other DJ errors that are occurring.

@lightalloy

This comment has been minimized.

Copy link
Collaborator

commented Apr 23, 2019

I'm closing this issue because the DeserializationErrors are fixed (at least most of them).
I'll open other issues for other kinds of DJ errors if they appear.

@lightalloy lightalloy closed this Apr 23, 2019

@coreyja coreyja referenced this issue Apr 28, 2019

Merged

Embed Twitch Live Streaming #2591

2 of 10 tasks complete

@lightalloy lightalloy referenced this issue Jun 12, 2019

Open

Move delay calls to ActiveJob #3136

2 of 12 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.