Usage: Fixup entity exists exception

[vishesh92]

Description

While creating helper records, it's possible that an event is processed twice which can result in failure due EntityExistsException. This PR adds a catch to ensure that job doesn't fail if the event is already present in the db.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

How did you try to break this feature and the system with this change?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 13.16%. Comparing base (9bd359a) to head (30556ab).

Additional details and impacted files

@@            Coverage Diff            @@
##               4.18    #8726   +/-   ##
=========================================
  Coverage     13.16%   13.16%           
  Complexity     9203     9203           
=========================================
  Files          2724     2724           
  Lines        258137   258137           
  Branches      40235    40235           
=========================================
  Hits          33989    33989           
+ Misses       219841   219840    -1     
- Partials       4307     4308    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[vishesh92]

@blueorangutan package

[blueorangutan]

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8817

[weizhouapache]

code lgtm

[GutoVeronezi]

I have never seem an event being processed twice without manual intervention in the database; instead, I already saw dup events, like NETWORK.OFFERING.REMOVE, NETWORK.OFFERING.ASSIGN, and VOLUME.DELETE, being persisted. Could the situation you are facing be similar to that? If it is the case, it would be insteresting if we could find the root cause.

@vishesh92, do you have the steps to reproduce the error?

[sureshanaparti]

clgtm

[sureshanaparti]

@blueorangutan test

[blueorangutan]

@sureshanaparti a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-9352)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 48383 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8726-t9352-kvm-centos7.zip
Smoke tests completed. 108 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_migrate_vm	Error	43.87	test_vm_life_cycle.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	Failure	1014.25	test_vpc_redundant.py

[vishesh92]

I have never seem an event being processed twice without manual intervention in the database; instead, I already saw dup events, like NETWORK.OFFERING.REMOVE, NETWORK.OFFERING.ASSIGN, and VOLUME.DELETE, being persisted. Could the situation you are facing be similar to that? If it is the case, it would be insteresting if we could find the root cause.

@vishesh92, do you have the steps to reproduce the error?

@GutoVeronezi I don't have any steps to reproduce the error. We saw the EntityExistsException in the log and it's probably because of duplicate events.

[DaanHoogland]

I have never seem an event being processed twice without manual intervention in the database; instead, I already saw dup events, like NETWORK.OFFERING.REMOVE, NETWORK.OFFERING.ASSIGN, and VOLUME.DELETE, being persisted. Could the situation you are facing be similar to that? If it is the case, it would be insteresting if we could find the root cause.
@vishesh92, do you have the steps to reproduce the error?

@GutoVeronezi I don't have any steps to reproduce the error. We saw the EntityExistsException in the log and it's probably because of duplicate events.

I think you just described the method of reproducing; duplicate any of the not yet processed events for create/start/register events.

[DaanHoogland]

code lgtm

[GutoVeronezi]

I have never seem an event being processed twice without manual intervention in the database; instead, I already saw dup events, like NETWORK.OFFERING.REMOVE, NETWORK.OFFERING.ASSIGN, and VOLUME.DELETE, being persisted. Could the situation you are facing be similar to that? If it is the case, it would be insteresting if we could find the root cause.
@vishesh92, do you have the steps to reproduce the error?

@GutoVeronezi I don't have any steps to reproduce the error. We saw the EntityExistsException in the log and it's probably because of duplicate events.

I think you just described the method of reproducing; duplicate any of the not yet processed events for create/start/register events.

My point here was to try to identify the workflows causing those duplicated events; until far, I only saw the exception when manually intervening in the database. However, if it an ACS' process that is causing this, it would be good if we could identify and fix it.

[DaanHoogland]

My point here was to try to identify the workflows causing those duplicated events; until far, I only saw the exception when manually intervening in the database. However, if it an ACS' process that is causing this, it would be good if we could identify and fix it.

That is a good point, but we haven't yet found way to reproduce it, It may be due to events being fired once and then handled twice, like you could expect in UDP connections, but nor reproduction scheme yet. This is symptom fighting admittedly.

[weizhouapache]

My point here was to try to identify the workflows causing those duplicated events; until far, I only saw the exception when manually intervening in the database. However, if it an ACS' process that is causing this, it would be good if we could identify and fix it.

ideally yes.

IMHO, sometimes it is difficult to reproduce an issue and find the root cause. Of so, we'd better apply the workaround to avoid subsequent issues.

[vishesh92]

@GutoVeronezi I spent some time but couldn't find the reason for duplication of events. Even if we find and fix the root cause, we might not be able to prevent such an occurrence again. So, it would be good to have the check on the usage server itself as well.

[GutoVeronezi]

@GutoVeronezi I spent some time but couldn't find the reason for duplication of events. Even if we find and fix the root cause, we might not be able to prevent such an occurrence again. So, it would be good to have the check on the usage server itself as well.

@vishesh92
Thanks for the effort; indeed, the warn message is better than the exception. Hope we can find the root cause in the future.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8903

[weizhouapache]

@blueorangutan test

[blueorangutan]

@weizhouapache a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-9439)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40460 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8726-t9439-kvm-centos7.zip
Smoke tests completed. 110 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File