Infinispan hotrod unable to add new server in IPv6 #10920

zapkub · 2023-05-10T03:32:41Z

zapkub
May 10, 2023

Hey guys, Anyone have an experience setting up ispn in IPv6 environment?
I have ispn cluster rebalanced properly but unable to connect to the cluster with hotrod client

Thanks for any help!

Edit:

For anyone looking for workaround. Please see my latest comment

Answered by tristantarrant

May 10, 2023

Sorry, I wrongly assumed that :: would be passed on as-is, but the JDK expands it to 0:0:0:0:0:0:0:0. I'm afraid that there is no workaround until those PRs are merged.

View full answer

tristantarrant · 2023-05-10T06:35:17Z

tristantarrant
May 10, 2023
Maintainer

The problem seems to be with the server, not with the client. How are you binding the server ? Can you enable trace logs for org.infinispan.server.hotrod and post the server logs here please ?

0 replies

zapkub · 2023-05-10T10:10:44Z

zapkub
May 10, 2023
Author

@tristantarrant big thanks for the response!

Here is the log and the configuration which cause the error ( I am running ispn with docker-compose IPv6 network)

I discovered that there is problem about getting the right IP address to bind with IPv6.

  infinispan1:
    image: infinispan/server:14.0.4.Final 
    command: ["--bind-address=::"] # here is bind with ipv6 wildcard
    environment:
      - NODE_NAME=is-1
      - _JAVA_OPTIONS=-Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true # prefer IPv6 and disable IPv4
      - JGROUPS_DISCOVERY_PROTOCOL=TCPIP
      - JGROUPS_DISCOVERY_PROPERTIES="initial_hosts=infinispan1[7800],infinispan2[7800]"
      - USER=admin
      - PASS=admin123
    volumes:
      - ./log4j2.xml:/opt/infinispan/server/conf/log4j2.xml
    ports:
      - 11222:11222
    networks:
      - ipv6-network

here is the ip addr result

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 # <<< LOOK LIKE IT BIND WITH THIS IP
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
461: eth0@if462: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:1c:00:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.28.0.5/16 brd 172.28.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2002:db8:a::5/64 scope global nodad <<< SHOULD BE THIS ONE INSTEAD
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe1c:5/64 scope link
       valid_lft forever preferred_lft forever

So I try to use binding network interface with <match-interface /> which then cause another kind of error which I will post in the next comment

And here is the log from org.infinispan.server.hotrod which host name of the server is weird

2023-05-10 09:59:58,003 TRACE (non-blocking-thread--p2-t28) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,004 TRACE (non-blocking-thread--p2-t28) [org.infinispan.server.hotrod.Encoder2x] Write response to EXEC messageId=230 status=Success prev=[B0x
2023-05-10 09:59:58,004 TRACE (non-blocking-thread--p2-t15) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=PING, version=40, messageId=231, cacheName='auth-sessions', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams={}}
2023-05-10 09:59:58,006 TRACE (non-blocking-thread--p2-t15) [org.infinispan.server.hotrod.Encoder2x] Check for partial topologies: endpoints={933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}, client-topology=-1, server-topology=5, server-client-topology=-101622776
2023-05-10 09:59:58,006 TRACE (non-blocking-thread--p2-t15) [org.infinispan.server.hotrod.Encoder2x] Send client topology update with topology id -101622776
2023-05-10 09:59:58,006 TRACE (non-blocking-thread--p2-t15) [org.infinispan.server.hotrod.Encoder2x] Topology cache contains: {933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}
2023-05-10 09:59:58,006 TRACE (non-blocking-thread--p2-t15) [org.infinispan.server.hotrod.Encoder2x] After read consistent hash filter, members are: {933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}
2023-05-10 09:59:58,006 TRACE (non-blocking-thread--p2-t15) [org.infinispan.server.hotrod.Encoder2x] Write hash distribution change response header HashDistAware20Response{topologyId=-101622776, numSegments=256, hashFunction=3}
2023-05-10 09:59:58,014 TRACE (non-blocking-thread--p2-t30) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH_MECH_LIST, version=31, messageId=234, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,014 TRACE (non-blocking-thread--p2-t30) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,014 TRACE (non-blocking-thread--p2-t30) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH, version=31, messageId=235, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,015 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,031 TRACE (non-blocking-thread--p2-t30) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH, version=31, messageId=236, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,032 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,033 TRACE (non-blocking-thread--p2-t30) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=PING, version=31, messageId=233, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,033 TRACE (non-blocking-thread--p2-t30) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,034 TRACE (non-blocking-thread--p2-t30) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=PREPARE_TX_2, version=40, messageId=232, cacheName='auth-sessions', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams={}}
2023-05-10 09:59:58,038 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.TransactionRequestProcessor] Transaction Xid{formatId=0, globalTransactionId=01,branchQualifier=01} is read only.
2023-05-10 09:59:58,039 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Check for partial topologies: endpoints={933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}, client-topology=-1, server-topology=5, server-client-topology=-101622776
2023-05-10 09:59:58,039 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Send client topology update with topology id -101622776
2023-05-10 09:59:58,040 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Topology cache contains: {933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}
2023-05-10 09:59:58,040 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] After read consistent hash filter, members are: {933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}
2023-05-10 09:59:58,040 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Write hash distribution change response header HashDistAware20Response{topologyId=-101622776, numSegments=256, hashFunction=3}
2023-05-10 09:59:58,053 TRACE (non-blocking-thread--p2-t32) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH_MECH_LIST, version=31, messageId=239, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,053 TRACE (non-blocking-thread--p2-t32) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,054 TRACE (non-blocking-thread--p2-t32) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH, version=31, messageId=240, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,055 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,075 TRACE (non-blocking-thread--p2-t32) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH, version=31, messageId=241, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,075 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,076 TRACE (non-blocking-thread--p2-t32) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=PING, version=31, messageId=238, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,076 TRACE (non-blocking-thread--p2-t32) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,076 TRACE (non-blocking-thread--p2-t32) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=GET_WITH_METADATA, version=40, messageId=237, cacheName='auth-sessions', flag=0, clientIntel=3, topologyId=-1, keyType=application/x-protostream, valueType=application/x-protostream, otherParams={}}
2023-05-10 09:59:58,078 TRACE (non-blocking-thread--p2-t32) [org.infinispan.server.hotrod.Encoder2x] Check for partial topologies: endpoints={933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}, client-topology=-1, server-topology=5, server-client-topology=-101622776
2023-05-10 09:59:58,078 TRACE (non-blocking-thread--p2-t32) [org.infinispan.server.hotrod.Encoder2x] Send client topology update with topology id -101622776
2023-05-10 09:59:58,078 TRACE (non-blocking-thread--p2-t32) [org.infinispan.server.hotrod.Encoder2x] Topology cache contains: {933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}
2023-05-10 09:59:58,078 TRACE (non-blocking-thread--p2-t32) [org.infinispan.server.hotrod.Encoder2x] After read consistent hash filter, members are: {933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}
2023-05-10 09:59:58,078 TRACE (non-blocking-thread--p2-t32) [org.infinispan.server.hotrod.Encoder2x] Write hash distribution change response header HashDistAware20Response{topologyId=-101622776, numSegments=256, hashFunction=3}
2023-05-10 09:59:58,342 TRACE (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH_MECH_LIST, version=31, messageId=244, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,342 TRACE (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,343 TRACE (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH, version=31, messageId=245, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,343 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,362 TRACE (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH, version=31, messageId=246, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,363 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,364 TRACE (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=PING, version=31, messageId=243, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,364 TRACE (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,364 TRACE (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=GET_WITH_METADATA, version=40, messageId=242, cacheName='auth-sessions', flag=0, clientIntel=3, topologyId=-1, keyType=application/x-protostream, valueType=application/x-protostream, otherParams={}}
2023-05-10 09:59:58,366 TRACE (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.Encoder2x] Check for partial topologies: endpoints={933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}, client-topology=-1, server-topology=5, server-client-topology=-101622776
2023-05-10 09:59:58,366 TRACE (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.Encoder2x] Send client topology update with topology id -101622776
2023-05-10 09:59:58,366 TRACE (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.Encoder2x] Topology cache contains: {933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}
2023-05-10 09:59:58,366 TRACE (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.Encoder2x] After read consistent hash filter, members are: {933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}
2023-05-10 09:59:58,366 TRACE (non-blocking-thread--p2-t2) [org.infinispan.server.hotrod.Encoder2x] Write hash distribution change response header HashDistAware20Response{topologyId=-101622776, numSegments=256, hashFunction=3}
2023-05-10 09:59:58,394 TRACE (non-blocking-thread--p2-t4) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH_MECH_LIST, version=31, messageId=249, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,394 TRACE (non-blocking-thread--p2-t4) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,395 TRACE (non-blocking-thread--p2-t4) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH, version=31, messageId=250, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,395 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,412 TRACE (non-blocking-thread--p2-t4) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH, version=31, messageId=251, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,412 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,413 TRACE (non-blocking-thread--p2-t4) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=PING, version=31, messageId=248, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,413 TRACE (non-blocking-thread--p2-t4) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,413 TRACE (non-blocking-thread--p2-t4) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=PREPARE_TX_2, version=40, messageId=247, cacheName='auth-sessions', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams={}}
2023-05-10 09:59:58,416 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.GlobalTxTable] [CacheXid{cacheName=auth-sessions, xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}}] Get TxState = null
2023-05-10 09:59:58,417 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.GlobalTxTable] [CacheXid{cacheName=auth-sessions, xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}}] Updating with function: org.infinispan.server.hotrod.tx.table.functions.CreateStateFunction@4eed995b
2023-05-10 09:59:58,436 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.GlobalTxTable] [CacheXid{cacheName=auth-sessions, xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}}] Return value is OK
2023-05-10 09:59:58,437 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.PerCacheTxTable] [Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}] New tx=TransactionImpl{xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}, status=ACTIVE}
2023-05-10 09:59:58,443 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.TransactionRequestProcessor] Key didn't exist for operation TransactionWrite{key=[B0x4A2433373164356533342D316338322D..[38],h=2ff129f6], versionRead=0, control=[NON_EXISTING], metadata=org.infinispan.metadata.EmbeddedMetadata$Builder@29ffdcc6, value=[B0x8201286B632E486F74526F64526F6F74..[1062],h=bac36c5c]}. Entry is null
2023-05-10 09:59:58,455 INFO  (blocking-thread--p3-t1) [org.hibernate.annotations.common.Version] HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
2023-05-10 09:59:58,665 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.GlobalTxTable] [CacheXid{cacheName=auth-sessions, xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}}] Updating with function: org.infinispan.server.hotrod.tx.table.functions.PreparingDecisionFunction@24b44c1b
2023-05-10 09:59:58,676 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.GlobalTxTable] [CacheXid{cacheName=auth-sessions, xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}}] Return value is OK
2023-05-10 09:59:58,681 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.GlobalTxTable] [CacheXid{cacheName=auth-sessions, xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}}] Updating with function: org.infinispan.server.hotrod.tx.table.functions.SetPreparedFunction@32d24aed
2023-05-10 09:59:58,683 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.GlobalTxTable] [CacheXid{cacheName=auth-sessions, xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}}] Return value is OK
2023-05-10 09:59:58,684 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Check for partial topologies: endpoints={933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}, client-topology=-1, server-topology=5, server-client-topology=-101622776
2023-05-10 09:59:58,684 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Send client topology update with topology id -101622776
2023-05-10 09:59:58,684 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Topology cache contains: {933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}
2023-05-10 09:59:58,684 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] After read consistent hash filter, members are: {933c39b372a5-58875=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}, b98599012fcf-11526=SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222}}
2023-05-10 09:59:58,684 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Write hash distribution change response header HashDistAware20Response{topologyId=-101622776, numSegments=256, hashFunction=3}
2023-05-10 09:59:58,704 TRACE (non-blocking-thread--p2-t6) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH_MECH_LIST, version=31, messageId=254, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,704 TRACE (non-blocking-thread--p2-t6) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,705 TRACE (non-blocking-thread--p2-t6) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH, version=31, messageId=255, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,705 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,723 TRACE (non-blocking-thread--p2-t6) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=AUTH, version=31, messageId=256, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,724 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,724 TRACE (non-blocking-thread--p2-t6) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=PING, version=31, messageId=253, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams=null}
2023-05-10 09:59:58,724 TRACE (non-blocking-thread--p2-t6) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change
2023-05-10 09:59:58,725 TRACE (non-blocking-thread--p2-t6) [org.infinispan.server.hotrod.BaseDecoder] Parsed header: HotRodHeader{op=COMMIT_TX, version=40, messageId=252, cacheName='', flag=0, clientIntel=3, topologyId=-1, keyType=application/unknown, valueType=application/unknown, otherParams={}}
2023-05-10 09:59:58,729 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.GlobalTxTable] [Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}] Set Transaction Decision to Commit
2023-05-10 09:59:58,735 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.GlobalTxTable] [Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}] Fetched CacheXids=[CacheXid{cacheName=auth-sessions, xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}}]
2023-05-10 09:59:58,741 TRACE (jgroups-6,b98599012fcf-11526) [org.infinispan.server.hotrod.tx.operation.CommitTransactionOperation] [Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}] Collected cache auth-sessions status OK
2023-05-10 09:59:58,742 TRACE (jgroups-6,b98599012fcf-11526) [org.infinispan.server.hotrod.tx.operation.BaseCompleteTransactionOperation] [Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}] Cache collected. Missing=0.
2023-05-10 09:59:58,742 TRACE (jgroups-6,b98599012fcf-11526) [org.infinispan.server.hotrod.tx.operation.BaseCompleteTransactionOperation] [Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}] All caches collected: [auth-sessions]
2023-05-10 09:59:58,749 TRACE (jgroups-6,b98599012fcf-11526) [org.infinispan.server.hotrod.tx.table.GlobalTxTable] [CacheXid{cacheName=auth-sessions, xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}}] Get TxState = TxState{globalTransaction=GlobalTransaction{id=22, addr=ClientAddress{localAddress=b98599012fcf-11526}, remote=false, xid=null, internalId=281479271677953}, status=MARK_COMMIT, modifications=[PutKeyValueCommand{key=WrappedByteArray[\J\$\3\7\1\d\5\e\3\4\-\1\c\8\2\-\4\a\e\7\-\b\8\e\1\-\f\4\2\6\b\9\0\5\0\3\a\2 (38 bytes)], value=ProtobufValueWrapper(length=1062, binary=[82 01 28 6B 63 2E 48 6F 74 52 6F 64 52 6F 6F 74 41 75 74 68 65 6E 74 69 63 61 74 69 6F 6E 53 65 73 73 69 6F 6E 45 6E 74...]), flags=[FORCE_WRITE_LOCK], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, returnEntry=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{version=NumericVersion{version=1}, lifespan=1799663, maxIdle=-1}, internalMetadata=null, successful=true, topologyId=5}], recoverable=true, timeout=300000, lastAccessTime=39516247065597}
2023-05-10 09:59:58,750 TRACE (jgroups-6,b98599012fcf-11526) [org.infinispan.server.hotrod.tx.operation.BaseCompleteTransactionOperation] [Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}] Completing local executed transaction.
2023-05-10 09:59:59,188 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.GlobalTxTable] [CacheXid{cacheName=auth-sessions, xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}}] Updating with function: org.infinispan.server.hotrod.tx.table.functions.SetCompletedTransactionFunction@45076c85
2023-05-10 09:59:59,191 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.GlobalTxTable] [CacheXid{cacheName=auth-sessions, xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}}] Return value is OK
2023-05-10 09:59:59,191 TRACE (blocking-thread--p3-t1) [org.infinispan.server.hotrod.tx.table.PerCacheTxTable] [Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}] Removed tx=TransactionImpl{xid=Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}, status=COMMITTED}
2023-05-10 09:59:59,191 TRACE (non-blocking-thread--p2-t8) [org.infinispan.server.hotrod.tx.operation.CommitTransactionOperation] [Xid{formatId=131077, globalTransactionId=20020DB8000A0000000000000000000600009A69645B6B1800000037717561726B7573,branchQualifier=20020DB8000A0000000000000000000600009A69645B6B18000000470000000000000000}] Sending reply 0
2023-05-10 09:59:59,192 TRACE (non-blocking-thread--p2-t8) [org.infinispan.server.hotrod.Encoder2x] Write topology response header with no change

5 replies

tristantarrant May 10, 2023
Maintainer

Apologies, it's a bug: the IPv6 wildcard we use to enable multi homed addresses is wrong.
Workaround for now is to use ["--bind-address=::0"]
I've created https://issues.redhat.com/browse/ISPN-14854

tristantarrant May 10, 2023
Maintainer

I've created PRs for main and 14.0.x: #10924 #10925

zapkub May 10, 2023
Author

getting strange. I try to use ::0 but the result of the address mode is still a single homed

2023-05-10 10:22:14,402 INFO  (main) [BOOT] JVM OpenJDK 64-Bit Server VM Red Hat, Inc. 17.0.5+8-LTS
2023-05-10 10:22:14,409 INFO  (main) [BOOT] JVM arguments = [-server, --add-exports, java.naming/com.sun.jndi.ldap=ALL-UNNAMED, -Xlog:gc*:file=/opt/infinispan/server/log/gc.log:time,uptimemillis:filecount=5,filesize=3M, -XX:+ExitOnOutOfMemoryError, -XX:MetaspaceSize=32m, -XX:MaxMetaspaceSize=96m, -Djava.net.preferIPv4Stack=true, -Djava.awt.headless=true, -Dvisualvm.display.name=infinispan-server, -Djava.util.logging.manager=org.infinispan.server.loader.LogManager, -Dinfinispan.server.home.path=/opt/infinispan, -classpath, :/opt/infinispan/boot/infinispan-server-runtime-14.0.4.Final-loader.jar, org.infinispan.server.loader.Loader, org.infinispan.server.Bootstrap, --bind-address=::0] # HERE IS THE BIND ADDRESS ARGS
2023-05-10 10:22:14,412 INFO  (main) [BOOT] PID = 185
2023-05-10 10:22:14,437 INFO  (main) [org.infinispan.SERVER] ISPN080000: Infinispan Server 14.0.4.Final starting
2023-05-10 10:22:14,437 INFO  (main) [org.infinispan.SERVER] ISPN080017: Server configuration: infinispan.xml
2023-05-10 10:22:14,437 INFO  (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan/server/conf/log4j2.xml
2023-05-10 10:22:14,953 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory'
2023-05-10 10:22:14,954 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory'
2023-05-10 10:22:14,958 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'iteration-filter-converter-factory'
2023-05-10 10:22:14,961 WARN  (main) [org.infinispan.SERVER] ISPN080059: No script engines are available
2023-05-10 10:22:15,599 WARN  (main) [org.infinispan.PERSISTENCE] ISPN000554: jboss-marshalling is deprecated and planned for removal
2023-05-10 10:22:15,626 INFO  (main) [org.infinispan.CONTAINER] ISPN000556: Starting user marshaller 'org.infinispan.commons.marshall.ImmutableProtoStreamMarshaller'
2023-05-10 10:22:16,438 INFO  (main) [org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel `cluster` with stack `tcp`
2023-05-10 10:22:16,446 INFO  (main) [org.jgroups.JChannel] local_addr: d8a02889-7ef2-4041-a64e-c70b5478a277, name: 59df59695d72-43833
2023-05-10 10:22:16,488 INFO  (main) [org.jgroups.protocols.FD_SOCK2] server listening on *.57800
2023-05-10 10:22:18,508 INFO  (main) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel cluster: [59df59695d72-43833|0] (1) [59df59695d72-43833]
2023-05-10 10:22:18,558 INFO  (main) [org.infinispan.CLUSTER] ISPN000079: Channel `cluster` local address is `59df59695d72-43833`, physical addresses are `[2002:db8:a:0:0:0:0:3%eth0:7800]`
2023-05-10 10:22:18,569 INFO  (main) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=14.0.4.Final timestamp=2023-05-10T10:22:18.568467193Z
2023-05-10 10:22:18,775 INFO  (main) [org.infinispan.CONTAINER] ISPN000104: Using EmbeddedTransactionManager
2023-05-10 10:22:19,054 INFO  (ForkJoinPool.commonPool-worker-2) [org.infinispan.server.core.telemetry.TelemetryServiceFactory] ISPN000953: OpenTelemetry integration is disabled
2023-05-10 10:22:19,067 INFO  (main) [org.infinispan.SERVER] ISPN080018: Started connector Resp (internal)
2023-05-10 10:22:19,071 DEBUG (ForkJoinPool.commonPool-worker-1) [org.infinispan.server.hotrod.HotRodServer] Externally facing address is null:-1
2023-05-10 10:22:19,080 DEBUG (ForkJoinPool.commonPool-worker-1) [org.infinispan.server.hotrod.HotRodServer] Map 59df59695d72-43833 cluster address with SingleHomedServerAddress{host='0:0:0:0:0:0:0:0', port=11222} server endpoint in address cache

tristantarrant May 10, 2023
Maintainer

Sorry, I wrongly assumed that :: would be passed on as-is, but the JDK expands it to 0:0:0:0:0:0:0:0. I'm afraid that there is no workaround until those PRs are merged.

Answer selected by zapkub

zapkub May 10, 2023
Author

I found the possible workaround for current version with a single binding address by pick the global scope IP address.

#!/bin/bash
GLOB_IP=$(ip addr | awk '/inet6 .* scope global.*/{sub(/\/[0-9]+$/,"",$2); print $2}')
/opt/infinispan/bin/launch.sh --bind-address=$GLOB_IP

Thank you for clarify the issue 👏

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Infinispan hotrod unable to add new server in IPv6 #10920

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Infinispan hotrod unable to add new server in IPv6 #10920

zapkub May 10, 2023

Replies: 2 comments · 5 replies

tristantarrant May 10, 2023 Maintainer

zapkub May 10, 2023 Author

tristantarrant May 10, 2023 Maintainer

tristantarrant May 10, 2023 Maintainer

zapkub May 10, 2023 Author

tristantarrant May 10, 2023 Maintainer

zapkub May 10, 2023 Author

zapkub
May 10, 2023

Replies: 2 comments 5 replies

tristantarrant
May 10, 2023
Maintainer

zapkub
May 10, 2023
Author

tristantarrant May 10, 2023
Maintainer

tristantarrant May 10, 2023
Maintainer

zapkub May 10, 2023
Author

tristantarrant May 10, 2023
Maintainer

zapkub May 10, 2023
Author