Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] AutoFollowIT testAutoFollowSoftDeletesDisabled failing #100510

Closed
edsavage opened this issue Oct 9, 2023 · 1 comment · Fixed by #101907
Closed

[CI] AutoFollowIT testAutoFollowSoftDeletesDisabled failing #100510

edsavage opened this issue Oct 9, 2023 · 1 comment · Fixed by #101907
Assignees
@DaveCTurner
Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI

Comments

@edsavage
Copy link
Contributor

edsavage commented Oct 9, 2023

Build scan:
https://gradle-enterprise.elastic.co/s/7ogsa7kjmbcp6/tests/:x-pack:plugin:ccr:internalClusterTest/org.elasticsearch.xpack.ccr.AutoFollowIT/testAutoFollowSoftDeletesDisabled
Reproduction line:

./gradlew ':x-pack:plugin:ccr:internalClusterTest' --tests "org.elasticsearch.xpack.ccr.AutoFollowIT.testAutoFollowSoftDeletesDisabled" -Dtests.seed=7380EBC21C6AE9 -Dtests.locale=be-BY -Dtests.timezone=Antarctica/Rothera -Druntime.java=21

Applicable branches:
7.17

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.ccr.AutoFollowIT&tests.test=testAutoFollowSoftDeletesDisabled
Failure excerpt:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1100, name=elasticsearch[follower2][generic][T#4], state=RUNNABLE, group=TGRP-AutoFollowIT]

  at __randomizedtesting.SeedInfo.seed([7380EBC21C6AE9:9F813802806BFB06]:0)

  Caused by: java.lang.AssertionError: java.lang.AssertionError: listener.onFailure failed

    at __randomizedtesting.SeedInfo.seed([7380EBC21C6AE9]:0)
    at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:71)
    at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38)
    at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)
    at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:115)
    at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100)
    at org.elasticsearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:149)
    at org.elasticsearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:147)
    at org.elasticsearch.transport.ClusterConnectionManager.failConnectionListener(ClusterConnectionManager.java:372)
    at org.elasticsearch.transport.ClusterConnectionManager.lambda$connectToNodeOrRetry$7(ClusterConnectionManager.java:249)
    at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144)
    at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)
    at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)
    at org.elasticsearch.action.support.ThreadedActionListener$2.doRun(ThreadedActionListener.java:110)
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.lang.Thread.run(Thread.java:1583)

    Caused by: java.lang.AssertionError: listener.onFailure failed

      at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:71)
      at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38)
      at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)
      at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:115)
      at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100)
      at org.elasticsearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:149)
      at org.elasticsearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:147)
      at org.elasticsearch.transport.ClusterConnectionManager.failConnectionListener(ClusterConnectionManager.java:372)
      at org.elasticsearch.transport.ClusterConnectionManager.lambda$connectToNodeOrRetry$7(ClusterConnectionManager.java:249)
      at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144)
      at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)
      at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)
      at org.elasticsearch.action.support.ThreadedActionListener$2.doRun(ThreadedActionListener.java:110)
      at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)
      at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
      at java.lang.Thread.run(Thread.java:1583)

      Caused by: java.lang.IllegalStateException: transport has been stopped

        at org.elasticsearch.transport.TcpTransport.ensureOpen(TcpTransport.java:899)
        at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:316)
        at org.elasticsearch.test.transport.StubbableTransport.openConnection(StubbableTransport.java:164)
        at org.elasticsearch.transport.ClusterConnectionManager.internalOpenConnection(ClusterConnectionManager.java:348)
        at org.elasticsearch.transport.ClusterConnectionManager.connectToNodeOrRetry(ClusterConnectionManager.java:199)
        at org.elasticsearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:98)
        at org.elasticsearch.transport.RemoteConnectionManager.connectToRemoteClusterNode(RemoteConnectionManager.java:64)
        at org.elasticsearch.transport.SniffConnectionStrategy$SniffClusterStateResponseHandler.handleNodes(SniffConnectionStrategy.java:450)
        at org.elasticsearch.transport.SniffConnectionStrategy$SniffClusterStateResponseHandler.access$300(SniffConnectionStrategy.java:418)
        at org.elasticsearch.transport.SniffConnectionStrategy$SniffClusterStateResponseHandler$1.onFailure(SniffConnectionStrategy.java:472)
        at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)
        at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38)
        at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:115)
        at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100)
        at org.elasticsearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:149)
        at org.elasticsearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:147)
        at org.elasticsearch.transport.ClusterConnectionManager.failConnectionListener(ClusterConnectionManager.java:372)
        at org.elasticsearch.transport.ClusterConnectionManager.lambda$connectToNodeOrRetry$7(ClusterConnectionManager.java:249)
        at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144)
        at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)
        at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)
        at org.elasticsearch.action.support.ThreadedActionListener$2.doRun(ThreadedActionListener.java:110)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.lang.Thread.run(Thread.java:1583)
@edsavage edsavage added :Distributed/CCR Issues around the Cross Cluster State Replication features >test-failure Triaged test failures from CI labels Oct 9, 2023
@elasticsearchmachine elasticsearchmachine added the Team:Distributed Meta label for distributed team label Oct 9, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@henningandersen henningandersen added the low-risk An open issue or test failure that is a low risk to future releases label Oct 11, 2023
@DaveCTurner DaveCTurner self-assigned this Nov 8, 2023
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Nov 8, 2023
@DaveCTurner
Fail listener on exception in TcpTransport#openConnection
d0f35c2 
Today `TcpTransport#openConnection` may throw exceptions on certain
kinds of failure, but other kinds of failure are passed to the listener.
This is trappy and not all callers handle it correctly. This commit
makes sure that all exceptions are passed to the listener.

Closes elastic#100510
@DaveCTurner DaveCTurner mentioned this issue Nov 8, 2023
Merged
DaveCTurner added a commit that referenced this issue Nov 9, 2023
@DaveCTurner
Fail listener on exception in TcpTransport#openConnection (#101907)
cefa750 
Today `TcpTransport#openConnection` may throw exceptions on certain
kinds of failure, but other kinds of failure are passed to the listener.
This is trappy and not all callers handle it correctly. This commit
makes sure that all exceptions are passed to the listener.

Closes #100510
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Nov 9, 2023
@DaveCTurner
Fail listener on exception in TcpTransport#openConnection (elastic#10…
94b2130 
…1907)

Today `TcpTransport#openConnection` may throw exceptions on certain
kinds of failure, but other kinds of failure are passed to the listener.
This is trappy and not all callers handle it correctly. This commit
makes sure that all exceptions are passed to the listener.

Closes elastic#100510
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Nov 9, 2023
@DaveCTurner
Fail listener on exception in TcpTransport#openConnection (elastic#10…
429b63a 
…1907)

Today `TcpTransport#openConnection` may throw exceptions on certain
kinds of failure, but other kinds of failure are passed to the listener.
This is trappy and not all callers handle it correctly. This commit
makes sure that all exceptions are passed to the listener.

Closes elastic#100510
@DaveCTurner DaveCTurner mentioned this issue Nov 9, 2023
Merged
elasticsearchmachine pushed a commit that referenced this issue Nov 9, 2023
@DaveCTurner
Fail listener on exception in TcpTransport#openConnection (#101907) (#…
3a7ea17 
…101954)

Today `TcpTransport#openConnection` may throw exceptions on certain
kinds of failure, but other kinds of failure are passed to the listener.
This is trappy and not all callers handle it correctly. This commit
makes sure that all exceptions are passed to the listener.

Closes #100510
elasticsearchmachine pushed a commit that referenced this issue Nov 9, 2023
@DaveCTurner
Fail listener on exception in TcpTransport#openConnection (#101907) (#…
e573c1d 
…101955)

Today `TcpTransport#openConnection` may throw exceptions on certain
kinds of failure, but other kinds of failure are passed to the listener.
This is trappy and not all callers handle it correctly. This commit
makes sure that all exceptions are passed to the listener.

Closes #100510
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DaveCTurner DaveCTurner

Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fail listener on exception in TcpTransport#openConnection DaveCTurner/elasticsearch
4 participants
@DaveCTurner @edsavage @henningandersen @elasticsearchmachine