c-s equiped with shard-aware dirver can fail with AssertionError that comes from closeAsync

[dkropachev]

When test is completed c-s closes connection to the cluster.
due to the bug in driver it can fail there with assertException:

java.lang.AssertionError
at com.datastax.driver.core.ConvictionPolicy$DefaultConvictionPolicy.signalConnectionClosed(ConvictionPolicy.java:90)
at com.datastax.driver.core.Connection.closeAsync(Connection.java:953)
at com.datastax.driver.core.HostConnectionPool.discardAvailableConnections(HostConnectionPool.java:882)
at com.datastax.driver.core.HostConnectionPool.closeAsync(HostConnectionPool.java:843)

[haaawk]

Do you have reproducer @dkropachev ? Is the error thrown on every cluster.close()?

[haaawk]

Also what version of the driver is this?

[dkropachev]

Do you have reproducer @dkropachev ? Is the error thrown on every cluster.close()?

Have no way to reproduce it.
This error is thrown once after which c-s goes to panic

Also what version of the driver is this?

No reliable information on that, test run where it happened is lost, but most likely it was scylla-driver-core-3.10.1-scylla-0-shaded

[dkropachev]

Most likely this exception comes from this:

When connection is closed it find host it is relates to by EndPoint in manager metadata,

src/main/java/com/datastax/driver/core/Connection.java:957  
    if (signaled.compareAndSet(false, true)) {
      Host host = getHost();
      if (host != null) {
        host.convictionPolicy.signalConnectionClosed(this);
      }
    }


src/main/java/com/datastax/driver/core/Connection.java:988  
  private Host getHost() {
    Metadata metadata = factory.manager.metadata;
    Host host = metadata.getHost(endPoint);
    // During init the host might not be in metatada.hosts yet, try the contact points
    if (host == null) {
      host = metadata.getContactPoint(endPoint);
    }
    return host;
  }


src/main/java/com/datastax/driver/core/Metadata.java:202
  Host getHost(EndPoint endPoint) {
    for (Host host : hosts.values()) {
      if (host.getEndPoint().equals(endPoint)) {
        return host;
      }
    }
    return null;
  }

There are two of three exiting Endpoit implementations that are overriding equals in a way that allows different instances of Endpoints to match:

src/main/java/com/datastax/driver/core/TranslatedAddressEndPoint.java:39
  public boolean equals(Object other) {
    if (other == this) {
      return true;
    } else if (other instanceof TranslatedAddressEndPoint) {
      TranslatedAddressEndPoint that = (TranslatedAddressEndPoint) other;
      return Objects.equal(this.translatedAddress, that.translatedAddress);
    } else {
      return false;
    }
  }

src/main/java/com/datastax/driver/core/SniEndPoint.java:80
  @Override
  public boolean equals(Object other) {
    if (other == this) {
      return true;
    } else if (other instanceof SniEndPoint) {
      SniEndPoint that = (SniEndPoint) other;
      return this.proxyAddress.equals(that.proxyAddress) && this.serverName.equals(that.serverName);
    } else {
      return false;
    }
  }

So, most likely this issue happens when connection was initialized on one host instance, and closed on another.

[haaawk]

Do you have reproducer @dkropachev ? Is the error thrown on every cluster.close()?

Have no way to reproduce it.
This error is thrown once after which c-s goes to panic

What does it mean for c-s to go panic? @dkropachev

[haaawk]

There are two of three exiting Endpoit implementations that are overriding equals in a way that allows different instances of Endpoints to match:

src/main/java/com/datastax/driver/core/TranslatedAddressEndPoint.java:39
  public boolean equals(Object other) {
    if (other == this) {
      return true;
    } else if (other instanceof TranslatedAddressEndPoint) {
      TranslatedAddressEndPoint that = (TranslatedAddressEndPoint) other;
      return Objects.equal(this.translatedAddress, that.translatedAddress);
    } else {
      return false;
    }
  }

src/main/java/com/datastax/driver/core/SniEndPoint.java:80
  @Override
  public boolean equals(Object other) {
    if (other == this) {
      return true;
    } else if (other instanceof SniEndPoint) {
      SniEndPoint that = (SniEndPoint) other;
      return this.proxyAddress.equals(that.proxyAddress) && this.serverName.equals(that.serverName);
    } else {
      return false;
    }
  }

They actually don't match according to the implementations of equals they provide.

[dkropachev]

Do you have reproducer @dkropachev ? Is the error thrown on every cluster.close()?

Have no way to reproduce it.
This error is thrown once after which c-s goes to panic

What does it mean for c-s to go panic? @dkropachev

It means that it immediately stops process execution due to the unhandled exception

[haaawk]

Do you have reproducer @dkropachev ? Is the error thrown on every cluster.close()?

Have no way to reproduce it.
This error is thrown once after which c-s goes to panic

What does it mean for c-s to go panic? @dkropachev

It means that it immediately stops process execution due to the unhandled exception

You mean it retries a request 10 times and after no success c-s fails - is that correct?

[dkropachev]

There are two of three exiting Endpoit implementations that are overriding equals in a way that allows different instances of Endpoints to match:

src/main/java/com/datastax/driver/core/TranslatedAddressEndPoint.java:39
  public boolean equals(Object other) {
    if (other == this) {
      return true;
    } else if (other instanceof TranslatedAddressEndPoint) {
      TranslatedAddressEndPoint that = (TranslatedAddressEndPoint) other;
      return Objects.equal(this.translatedAddress, that.translatedAddress);
    } else {
      return false;
    }
  }

src/main/java/com/datastax/driver/core/SniEndPoint.java:80
  @Override
  public boolean equals(Object other) {
    if (other == this) {
      return true;
    } else if (other instanceof SniEndPoint) {
      SniEndPoint that = (SniEndPoint) other;
      return this.proxyAddress.equals(that.proxyAddress) && this.serverName.equals(that.serverName);
    } else {
      return false;
    }
  }

They actually don't match according to the implementations of equals they provide.

Not quite sure what it means.

[haaawk]

There are two of three exiting Endpoit implementations that are overriding equals in a way that allows different instances of Endpoints to match:

src/main/java/com/datastax/driver/core/TranslatedAddressEndPoint.java:39
  public boolean equals(Object other) {
    if (other == this) {
      return true;
    } else if (other instanceof TranslatedAddressEndPoint) {
      TranslatedAddressEndPoint that = (TranslatedAddressEndPoint) other;
      return Objects.equal(this.translatedAddress, that.translatedAddress);
    } else {
      return false;
    }
  }

src/main/java/com/datastax/driver/core/SniEndPoint.java:80
  @Override
  public boolean equals(Object other) {
    if (other == this) {
      return true;
    } else if (other instanceof SniEndPoint) {
      SniEndPoint that = (SniEndPoint) other;
      return this.proxyAddress.equals(that.proxyAddress) && this.serverName.equals(that.serverName);
    } else {
      return false;
    }
  }

They actually don't match according to the implementations of equals they provide.

I don't think that's the problem. Since getHost compares using equals and different implementations of EndPoint never equal.

[dkropachev]

Do you have reproducer @dkropachev ? Is the error thrown on every cluster.close()?

Have no way to reproduce it.
This error is thrown once after which c-s goes to panic

What does it mean for c-s to go panic? @dkropachev

It means that it immediately stops process execution due to the unhandled exception

You mean it retries a request 10 times and after no success c-s fails - is that correct?

No, it stop execution right away, at least in the case that is reffed here

[haaawk]

Do you have reproducer @dkropachev ? Is the error thrown on every cluster.close()?

Have no way to reproduce it.
This error is thrown once after which c-s goes to panic

What does it mean for c-s to go panic? @dkropachev

It means that it immediately stops process execution due to the unhandled exception

You mean it retries a request 10 times and after no success c-s fails - is that correct?

No, it stop execution right away, at least in the case that is reffed here

What's the exception that causes it to stop?