Add a 'to date' boundary to batch jobs #17
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
👍 (code review) |
Currently, the batch jobs have a 'from date' boundary only, which is incremented each time a new batch job is created. This may cause an issue when many batch jobs are created and don't have the time to be processed: the same records will be imported several times. Example with 4 batch jobs with an interval of 10 minutes (the 30 seconds difference is a buffer because Magento may commit a new record 'created_at' with a delay) : Batch job 1 from 2014-07-17 10:00:00 Batch job 2 from 2014-07-17 10:09:30 Batch job 3 from 2014-07-17 10:19:00 Batch job 4 from 2014-07-17 10:28:30 A product A is modified on Magento at 2014-07-17 10:25:00 None of the batch job could been executed so far (maybe an interruption or a high load). When the batch job 1 will be processed, it will see that product A has been modified later than 10:00 so it creates an import job. Turn of batch job 2, it sees the product modified after 10:10, creates an import job... and so on. This patch adds a 'to' boundary so the batch jobs become: Batch job 1 from 2014-07-17 10:00:00 to 2014-07-17 10:10:00 Batch job 2 from 2014-07-17 10:09:30 to 2014-07-17 10:19:30 Batch job 3 from 2014-07-17 10:19:00 to 2014-07-17 10:19:00 Batch job 4 from 2014-07-17 10:28:30 to 2014-07-17 10:28:30 Thus, in the same situation than before, only the batch job 3 would see the record.
|
Rebased |
|
👍 |
| and returns a list of ids | ||
|
|
||
| :rtype: list | ||
| """ | ||
| if filters is None: | ||
| filters = {} | ||
| dt_fmt = '%Y/%m/%d %H:%M:%S' |
There was a problem hiding this comment.
Can this be factored out as a global variable in __init__.py , maybe?
There was a problem hiding this comment.
You are right, done (put in backend_adapter.py which is where the things related to the Magento API lie).
guewen
force-pushed
the
7.0-add-boundary-to-batch-jobs
branch
from
3deab3f
to
53c1edc
Compare
|
👍 |
guewen
added a commit
that referenced
this pull request
Sep 17, 2014
Add a 'to date' boundary to batch jobs
jcoux
pushed a commit
to camptocamp/connector-magento
that referenced
this pull request
May 14, 2019
…rement Add multiple cancel procurement
saxomanu
pushed a commit
to saxomanu/connector-magento
that referenced
this pull request
Jun 5, 2020
[FIX] double call for export tracking
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, the batch jobs have a 'from date' boundary only, which is
incremented each time a new batch job is created.
This may cause an issue when many batch jobs are created and don't have
the time to be processed: the same records will be imported several
times.
Example with 4 batch jobs with an interval of 10 minutes (the 30 seconds
difference is a buffer because Magento may commit a new record 'created_at' with a
delay) :
Batch job 1 from 2014-07-17 10:00:00
Batch job 2 from 2014-07-17 10:09:30
Batch job 3 from 2014-07-17 10:19:00
Batch job 4 from 2014-07-17 10:28:30
A product A is modified on Magento at 2014-07-17 10:25:00
None of the batch job could been executed so far (maybe an interruption
or a high load). When the batch job 1 will be processed, it will see
that product A has been modified later than 10:00 so it creates an
import job. Turn of batch job 2, it sees the product modified after
10:10, creates an import job... and so on.
This patch adds a 'to' boundary so the batch jobs become:
Batch job 1 from 2014-07-17 10:00:00 to 2014-07-17 10:10:00
Batch job 2 from 2014-07-17 10:09:30 to 2014-07-17 10:19:30
Batch job 3 from 2014-07-17 10:19:00 to 2014-07-17 10:19:00
Batch job 4 from 2014-07-17 10:28:30 to 2014-07-17 10:28:30
Thus, in the same situation than before, only the batch job 3 would see
the record.