Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crontab not saving records #4265

Closed
Willbv opened this issue May 15, 2019 · 3 comments

Comments

Projects
None yet
2 participants
@Willbv
Copy link

commented May 15, 2019

I am running some daily imports for some products / prices and have set these up in crontab to do this automatically, recently however this doesn't seem to be updating the record prices although it does set them to 0.00 when that field is nullable and has no default value. Yet when I run the exact same command from the server as the same user it works and then furthermore after manually run and only then the crontab will update this if the data file changes.

  1. Import products via crontab early in the morning (roughly 3900 records)
  2. Import Prices by finding the product record and then attaching & saving the record

No errors occur during this and I put in some debugging to see what was going on,

  1. The price file has the correct price and finds the corresponding product record through Record::model()->findByAttributes(xxx)
  2. The record gets set the correct price
  3. The record successfully saves Record->save()
  4. Immediately after within the same command if you query the database the record price is 0.00
  • Craft version: 2.6.3019
  • PHP version: 7.0.33
  • Database driver & version: Ver 14.14 Distrib 5.7.26
  • Plugins & versions: All in custom plugin
@Willbv

This comment has been minimized.

Copy link
Author

commented May 17, 2019

Have been adding some more debugging to this and adjusting how its handled to see if I can get anything to work.

I originally had been looping through the price data and finding the product record individually to update and save but have reworked it to get all relevant records first then loop through those to match up prices then save.
Still getting the same issue of 0.00 in the database, but after the records are save in the same function I have a database query to get all products and report any with 0.00 price and one of the two new products/prices added the night previous had the correct price in this query (the other was 0.00 like most of them) yet still ends up with no price in the database at a later point.

Again like always manually running the exact same command from the same user on the server runs fine and gets everything updated as expected.

@Willbv

This comment has been minimized.

Copy link
Author

commented May 20, 2019

This appears to be a conflict with imports on the same table and will confirm on next import.

There's a large import at 2AM for the prices and a frequent import every 5 minutes to keep the stock level up to date. I think when you get the records in the Record::model()->findAll() format in two locations at the same time there are conflicts in saving them while they each have unique updates, which explains why running manually works because its rare to run exactly on the 5 min interval.

Additionally this morning when manually updating a large import of ~300 new products the last 7 didn't update the price as expected, they were all updated at 8:00:11 where as the ones previous were <= 8:00:10 and corresponds with a conflict of updating records.

I have disallowed the 5min import for the hours where big imports relate to the same table to see if this fixes the issue, not sure if this is something that can be addressed or really a bug any more, rather just a rare case but expected functionality.

@angrybrad

This comment has been minimized.

Copy link
Member

commented May 20, 2019

Definitely sounds like some sort of race condition. Might want to look at adding some locking/mutex support to your logic to minimize the chances of one: https://www.yiiframework.com/doc/api/2.0/yii-mutex-mutex

@angrybrad angrybrad closed this May 20, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.