Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestEntitlement CI Problem #1694

Closed
xsalefter opened this issue May 17, 2022 · 4 comments
Closed

TestEntitlement CI Problem #1694

xsalefter opened this issue May 17, 2022 · 4 comments

Comments

@xsalefter
Copy link
Contributor

xsalefter commented May 17, 2022 •
edited

Sometimes in CI, TestEntitlement#testCreateSubscriptionBillingInThePast() has assertion error (that will described below). This is the first PR that encounter this problem.

But when run this test in local using IDE, TestEntitlement#testCreateSubscriptionBillingInThePast() result is Ok and consistent without any problem. Running TestEntitlement in terminal using mvn test -Dorg.killbill.billing.dbi.test.mysql=true -Dtest="TestEntitlement" command is Ok without problem.

On the other side (but not sure if this is related to the main problem), running TestEntitlement test suite using IDE locally will lead to failed test in TestEntitlement#testSubscriptionEventsWithCatalogChange() (Log in comment)

Local test done both in linux (ubuntu) and macOS, using OpenJDK 11.

TestEntitlement#testCreateSubscriptionBillingInThePast() log:

....
2022-05-17T07:57:25.4286915Z [INFO] 
2022-05-17T07:57:25.4287389Z [INFO] Results:
2022-05-17T07:57:25.4287914Z [INFO] 
2022-05-17T07:57:25.4290962Z [ERROR] Failures: 
2022-05-17T07:57:25.4291893Z [ERROR]   TestEntitlement>GuicyKillbillTestSuite.run:208->testCreateSubscriptionBillingInThePast:871 
expected 
[class Subscription {
     org.killbill.billing.client.model.gen.Subscription@dd8820f
     accountId: 63ed1e1d-b583-43e9-9123-fed802812c35
     bundleId: 9eb3e6b5-c34a-4322-95ea-8d16e556fb25
     bundleExternalKey: 9eb3e6b5-c34a-4322-95ea-8d16e556fb25
     subscriptionId: 75f4a971-67df-48a3-977a-a363b23e21e5
     externalKey: 75f4a971-67df-48a3-977a-a363b23e21e5
     startDate: 2012-04-25T00:03:42.000Z
     productName: Shotgun
     productCategory: BASE
     billingPeriod: MONTHLY
     phaseType: EVERGREEN
     priceList: DEFAULT
     planName: shotgun-monthly
     state: ACTIVE
     sourceType: NATIVE
     cancelledDate: null
     chargedThroughDate: 2012-03-25
     billingStartDate: 2012-03-25T00:03:42.000Z
     billingEndDate: null
     billCycleDayLocal: 24
     events: [class EventSubscription {
         org.killbill.billing.client.model.gen.EventSubscription@fa195e2d
         eventId: 4799b6a1-6ca0-40f1-ac84-1a21d635e8bf
         billingPeriod: MONTHLY
         effectiveDate: 2012-03-25T00:03:42.000Z
         plan: shotgun-monthly
         product: Shotgun
         priceList: DEFAULT
         eventType: START_BILLING
         isBlockedBilling: false
         isBlockedEntitlement: false
         serviceName: billing-service
         serviceStateName: START_BILLING
         phase: shotgun-monthly-trial
         auditLogs: []
     }, class EventSubscription {
         org.killbill.billing.client.model.gen.EventSubscription@60d72475
         eventId: 5991af2f-9e4f-4a2d-9a78-607217e54d2d
         billingPeriod: MONTHLY
         effectiveDate: 2012-04-24T00:03:42.000Z
         plan: shotgun-monthly
         product: Shotgun
         priceList: DEFAULT
         eventType: PHASE
         isBlockedBilling: false
         isBlockedEntitlement: false
         serviceName: entitlement+billing-service
         serviceStateName: PHASE
         phase: shotgun-monthly-evergreen
         auditLogs: []
     }, class EventSubscription {
         org.killbill.billing.client.model.gen.EventSubscription@2ee2b863
         eventId: d41683dd-86ae-4478-87d2-c598e082c0b6
         billingPeriod: MONTHLY
         effectiveDate: 2012-04-25T00:03:42.000Z
         plan: shotgun-monthly
         product: Shotgun
         priceList: DEFAULT
         eventType: START_ENTITLEMENT
         isBlockedBilling: false
         isBlockedEntitlement: false
         serviceName: entitlement-service
         serviceStateName: ENT_STARTED
         phase: shotgun-monthly-evergreen
         auditLogs: []
     }]
     priceOverrides: null
     prices: [class PhasePrice {
         planName: shotgun-monthly
         phaseName: shotgun-monthly-trial
         phaseType: TRIAL
         fixedPrice: 0
         recurringPrice: null
         usagePrices: []
     }, class PhasePrice {
         planName: shotgun-monthly
         phaseName: shotgun-monthly-evergreen
         phaseType: EVERGREEN
         fixedPrice: null
         recurringPrice: 249.95
         usagePrices: []
     }]
     auditLogs: []
 }] but found 
 [class Subscription {
     org.killbill.billing.client.model.gen.Subscription@f694af38
     accountId: 63ed1e1d-b583-43e9-9123-fed802812c35
     bundleId: 9eb3e6b5-c34a-4322-95ea-8d16e556fb25
     bundleExternalKey: 9eb3e6b5-c34a-4322-95ea-8d16e556fb25
     subscriptionId: 75f4a971-67df-48a3-977a-a363b23e21e5
     externalKey: 75f4a971-67df-48a3-977a-a363b23e21e5
     startDate: 2012-04-25T00:03:42.000Z
     productName: Shotgun
     productCategory: BASE
     billingPeriod: MONTHLY
     phaseType: EVERGREEN
     priceList: DEFAULT
     planName: shotgun-monthly
     state: ACTIVE
     sourceType: NATIVE
     cancelledDate: null
     chargedThroughDate: 2012-05-24
     billingStartDate: 2012-03-25T00:03:42.000Z
     billingEndDate: null
     billCycleDayLocal: 24
     events: [class EventSubscription {
         org.killbill.billing.client.model.gen.EventSubscription@fa195e2d
         eventId: 4799b6a1-6ca0-40f1-ac84-1a21d635e8bf
         billingPeriod: MONTHLY
         effectiveDate: 2012-03-25T00:03:42.000Z
         plan: shotgun-monthly
         product: Shotgun
         priceList: DEFAULT
         eventType: START_BILLING
         isBlockedBilling: false
         isBlockedEntitlement: false
         serviceName: billing-service
         serviceStateName: START_BILLING
         phase: shotgun-monthly-trial
         auditLogs: []
     }, class EventSubscription {
         org.killbill.billing.client.model.gen.EventSubscription@60d72475
         eventId: 5991af2f-9e4f-4a2d-9a78-607217e54d2d
         billingPeriod: MONTHLY
         effectiveDate: 2012-04-24T00:03:42.000Z
         plan: shotgun-monthly
         product: Shotgun
         priceList: DEFAULT
         eventType: PHASE
         isBlockedBilling: false
         isBlockedEntitlement: false
         serviceName: entitlement+billing-service
         serviceStateName: PHASE
         phase: shotgun-monthly-evergreen
         auditLogs: []
     }, class EventSubscription {
         org.killbill.billing.client.model.gen.EventSubscription@2ee2b863
         eventId: d41683dd-86ae-4478-87d2-c598e082c0b6
         billingPeriod: MONTHLY
         effectiveDate: 2012-04-25T00:03:42.000Z
         plan: shotgun-monthly
         product: Shotgun
         priceList: DEFAULT
         eventType: START_ENTITLEMENT
         isBlockedBilling: false
         isBlockedEntitlement: false
         serviceName: entitlement-service
         serviceStateName: ENT_STARTED
         phase: shotgun-monthly-evergreen
         auditLogs: []
     }]
     priceOverrides: null
     prices: [class PhasePrice {
         planName: shotgun-monthly
         phaseName: shotgun-monthly-trial
         phaseType: TRIAL
         fixedPrice: 0
         recurringPrice: null
         usagePrices: []
     }, class PhasePrice {
         planName: shotgun-monthly
         phaseName: shotgun-monthly-evergreen
         phaseType: EVERGREEN
         fixedPrice: null
         recurringPrice: 249.95
         usagePrices: []
     }]
     auditLogs: []
 }]
....
@xsalefter
Copy link
Contributor Author

TestEntitlement#testSubscriptionEventsWithCatalogChange() error message:

java.lang.AssertionError
	at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.read(ResponseSubscribers.java:398)
	at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
	at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
	at org.killbill.billing.client.KillBillHttpClient.unmarshalResponse(KillBillHttpClient.java:462)
	at org.killbill.billing.client.KillBillHttpClient.deserializeResponse(KillBillHttpClient.java:410)
	at org.killbill.billing.client.KillBillHttpClient.doPrepareRequestInternal(KillBillHttpClient.java:350)
	at org.killbill.billing.client.KillBillHttpClient.doPrepareRequest(KillBillHttpClient.java:296)
	at org.killbill.billing.client.KillBillHttpClient.doGet(KillBillHttpClient.java:264)
	at org.killbill.billing.client.KillBillHttpClient.doGet(KillBillHttpClient.java:259)
	at org.killbill.billing.client.api.gen.CatalogApi.getCatalogXml(CatalogApi.java:191)
	at org.killbill.billing.jaxrs.TestJaxrsBase.uploadTenantCatalog(TestJaxrsBase.java:440)
	at org.killbill.billing.jaxrs.TestEntitlement.testSubscriptionEventsWithCatalogChange(TestEntitlement.java:1353)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:238)
	at org.killbill.billing.GuicyKillbillTestSuite.run(GuicyKillbillTestSuite.java:208)
	at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:252)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:595)
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
	at org.testng.TestRunner.privateRun(TestRunner.java:764)
	at org.testng.TestRunner.run(TestRunner.java:585)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
	at org.testng.TestNG.runSuites(TestNG.java:1069)
	at org.testng.TestNG.run(TestNG.java:1037)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)

@xsalefter xsalefter mentioned this issue May 17, 2022
Merged
@xsalefter
Copy link
Contributor Author

IMO the problem is lied in line ~872 : Assert.assertEquals(subscription, entitlementJson);.

Lets take a look at other assertion in TestEntitlement. If I understand TestEntitlement#verifyChargedThroughDate() correctly, it used to get up-to-date Subscription's chargedThroughDate value. In test case, entitlementJson.chargedThroughDate() value still 2012-03-24, where the "correct" value is 2012-05-24.

In short, in snippet below, there's no guarantee that,

final Subscription entitlementJson = subscriptionApi.createSubscription( ... ) // line ~849

The entitlementJson will return up-to-date or expected chargedThroughDate value.

With this in mind, back to first sentence of this comment, assertion in this line:

Assert.assertEquals(subscription, entitlementJson);

will not always equal, because equals() method in Subscription.java (killbill-client-java) add chargedThroughDate to comparison (Subscription.java line 303).

This is the diff of those 2 objects: https://www.diffchecker.com/uHlaknex

@pierre
Copy link
Member

pierre commented May 18, 2022

Yup, you're right! This comment explains it actually:

killbill/profiles/killbill/src/test/java/org/killbill/billing/jaxrs/TestEntitlement.java

Line 1419 in ea72fab

// The call completion may return after the INVOICE event was received and prior the CTD was updated as it

I would remove Assert.assertEquals(subscription, entitlementJson) and replace it with a series of assertEquals statements on individual fields (all but the CTD).

@xsalefter
Copy link
Contributor Author

Fixed by this commit: 081c08c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pierre @xsalefter