Slack failing CI build

[sherifnada]

Expected Behavior

Slack's CI build has been failing repeatedly. We should investigate and fix. See the Connector health page for more information.

2021-02-04 12:38:15 INFO i.a.c.i.LineGobbler(voidCall):69 - *** - parsing
2021-02-04 12:38:15 INFO i.a.c.i.LineGobbler(voidCall):69 - *** - executing command TestAction.TEARDOWN
2021-02-04 12:38:15 INFO i.a.c.i.LineGobbler(voidCall):69 - *** - writing output
2021-02-04 12:38:15 INFO i.a.c.i.LineGobbler(voidCall):69 - *** - complete

Failures (3):
  JUnit Jupiter:PythonTestSource:testIdenticalFullRefreshes()
    MethodSource [className = 'io.airbyte.integrations.standardtest.source.PythonTestSource', methodName = 'testIdenticalFullRefreshes', methodParameterTypes = '']
    => org.opentest4j.AssertionFailedError: Expected two full refresh syncs to produce the same records ==> expected: <449> but was: <448>
       org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
       org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
       org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:542)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.assertSameRecords(StandardSourceTest.java:445)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.testIdenticalFullRefreshes(StandardSourceTest.java:286)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       java.base/java.lang.reflect.Method.invoke(Method.java:564)
       [...]
  JUnit Jupiter:PythonTestSource:testIncrementalSyncWithState()
    MethodSource [className = 'io.airbyte.integrations.standardtest.source.PythonTestSource', methodName = 'testIncrementalSyncWithState', methodParameterTypes = '']
    => org.opentest4j.AssertionFailedError: Expected incremental sync to produce STATE messages ==> expected: <false> but was: <true>
       org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
       org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:40)
       org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:218)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.testIncrementalSyncWithState(StandardSourceTest.java:323)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       java.base/java.lang.reflect.Method.invoke(Method.java:564)
       org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
       org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
       [...]
  JUnit Jupiter:PythonTestSource:testEmptyStateIncrementalIdenticalToFullRefresh()
    MethodSource [className = 'io.airbyte.integrations.standardtest.source.PythonTestSource', methodName = 'testEmptyStateIncrementalIdenticalToFullRefresh', methodParameterTypes = '']
    => org.opentest4j.AssertionFailedError: Expected a full refresh sync and incremental sync with no input state to produce identical records ==> expected: <449> but was: <443>
       org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
       org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
       org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:542)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.assertSameRecords(StandardSourceTest.java:445)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.testEmptyStateIncrementalIdenticalToFullRefresh(StandardSourceTest.java:362)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       java.base/java.lang.reflect.Method.invoke(Method.java:564)
       [...]

Test run finished after 31742 ms
[         2 containers found      ]
[         0 containers skipped    ]
[         2 containers started    ]
[         0 containers aborted    ]
[         2 containers successful ]
[         0 containers failed     ]
[         7 tests found           ]
[         0 tests skipped         ]
[         7 tests started         ]
[         0 tests aborted         ]
[         4 tests successful      ]
[         3 tests failed          ]

There are failing tests. See https://docs.airbyte.io/contributing-to-airbyte/building-new-connector/standard-source-tests for more information about the standard source test suite.


> Task :airbyte-integrations:connectors:source-slack-singer:standardSourceTestPython FAILED
FAILURE: Build failed with an exception.

• Check the correct operation of the Incremental for the connector.
• Check data for streams called during CI tests.

[sherifnada]

We're receiving different users when pulling data from the users stream -- unclear why. Still need to root cause.

[sherifnada]

let's timebox to 1d and see what we can find.