-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Exception - Cannot insert duplicate key #2
Comments
I think there may be more than one bug here. First, the Second, I am wondering if the truncate is working correctly. You have already confirmed that there is only one record with this sys_id. If the database table is truncated, then it should be empty when the load starts, so the PK violation should not occur. Is it possible for you to verify that |
Here is another possibility: I noticed that this job had been running for more than an hour before it failed. Is it possible that a record was inserted or deleted in the If the issue is background inserts, then it should be possible to correct for it by adding |
There is insert/update activity occurring at the same time as my extract job is running. Unfortunately, there's not much I can do about that.
I tried manually truncating the table before running, and adding Interestingly, the second occurrence of the key is appearing nowhere close to the first - making me think this isn't just a shift in paging from insert/delete causing it to appear in two adjacent intervals.
I'll give this a try next
|
(1) Would partitioning help? Yes. This condition can still occur if you partition, but only if the record gets inserted/deleted in the partition that is currently being processed. Since the partitions are processed in reverse chronological order, the likelihood drops sharply after you get past the first partition. |
There are some changes in Beta 8 which may help with this issue. Please see the notes here: If you need to load a large table that is being concurrently updated, I recommend splitting it into pieces. For example # companies created before March 1 - name: core_company_before_20180301 source: core_company created: [void,2018-03-01] partition: month orderby: sysid # companies with null sys_created_on - name: core_company_null_created source: core_company created: [void,void] filter: sys_created_onISEMPTY # companies created or updated since March 1 - name: core_company_after_20180301 source: core_company since: 2018-03-01 orderby: sysid Let me know if the |
The latest release uses a default You may also want to test out the Thanks again. |
Not sure if there's a bug or a weird data condition; seemed worth reporting though.
For reference, the record in the error was created and last updated days ago (not in the middle of the load)
Property file
Load Configuration
Command
java -ea -jar sndml-3.0.1-b.7-mssql.jar -p property.prpty -y load.yaml &> 20180227.05.log
The text was updated successfully, but these errors were encountered: