New peers in a crdt cluster don't receive state #784

NatoBoram · 2019-05-20T03:31:09Z

Additional information:

OS: Ubuntu
IPFS Cluster version: 0.10.1+git9692e368f3e1a4f5eec224dd165822550e28c7a0
Installation method: built from source

Describe the bug:

New clusters initialized from this version don't receive the current state from other peers. ipfs-cluster-ctl status on new members returns nothing. Pinning a CID then running ipfs-cluster-ctl status does return the correct result, but that new CID doesn't appear on other peers' status.

Other peers are shown in ipfs-cluster-ctl peers ls.

The text was updated successfully, but these errors were encountered:

hsanjuan · 2019-05-20T08:22:48Z

You would need to wait a little bit until someone pins something or re-broadcast is triggered (every minute by default, controlled by rebroadcast_interval).

NatoBoram · 2019-05-20T10:02:08Z

After a full night and some manual pins added, I can confirm that the state is not transferred to the new peers.

ipfs-cluster-ctl peers ls :

Sees 2 other peers

ipfs-cluster-ctl pins ls and ipfs-cluster-ctl status :

NatoBoram · 2019-05-20T17:58:28Z

I added the node responsible for pinning in everyone's trusted_peers, and now the state was received. However, ipfs-cluster-ctl status will only display yourself and peers that trust you. Looks good to me.

I think it would be appropriate to consider the bootstrap target a trusted_peers regardless of config. It would also make sense to have your own trusted_peers be propagated to those who trust you.

hsanjuan · 2019-05-20T18:16:09Z

I added the node responsible for pinning in everyone's trusted_peers, and now the state was received.

Ah indeed! That was pretty obvious. At this point you are testing things that I haven't had time to setup myself.

ipfs-cluster-ctl status will only display yourself and peers that trust you

Yes, if a peer does not trust you it won't let you sniff around or touch anything.

I think it would be appropriate to consider the bootstrap target a trusted_peers regardless of config. It would also make sense to have your own trusted_peers be propagated to those who trust you.

The plan is to make configuration template, so that any peer can start with something like ipfs-cluster-service --config /ipfs/Qm... and that template will have the trusted_peers and the bootstrap correctly configured.

Auto-configuring the bootstrap peer(s) as "trusted" is probably the intended behaviour for 95% of cases, but technically you should be able to bootstrap to anyone (trusted or not).

NatoBoram · 2019-05-21T03:46:11Z

I could reproduce this issue on another computer. I have 3 peers bootstrapped to another one, and one of those 3 isn't receiving the state. They all have the exact same configurations and versions of IPFS and IPFS-Cluster.

Here's all the info about the defective peer.

System

OS: elementary OS 5.0 Juno x86_64 
Host: S551LN 1.0 
Kernel: 4.15.0-51-generic 
Uptime: 54 mins 
Packages: 1897 
Shell: bash 4.4.19 
Terminal: /dev/pts/0 
CPU: Intel i7-4500U (4) @ 3.000GHz 
GPU: NVIDIA GeForce 840M 
GPU: Intel Haswell-ULT 
Memory: 1037MiB / 7860MiB

IPFS

go-ipfs version: 0.4.21-rc1-cf3b6c43d
Repo version: 7
System version: amd64/linux
Golang version: go1.12.5

IPFS-Cluster

0.10.1+git30ba6f82dd8050e4df302f162ac29b3a4f4f8715

Peers

12D3KooWCWXtr2t7uLRdzwX9CitfLi5WApSbC4jY812NXdcRkWj4 | hp-server | Sees 3 other peers
  > Addresses:
    - /ip4/127.0.0.1/tcp/9096/ipfs/12D3KooWCWXtr2t7uLRdzwX9CitfLi5WApSbC4jY812NXdcRkWj4
    - /ip4/135.23.195.89/tcp/35218/ipfs/12D3KooWCWXtr2t7uLRdzwX9CitfLi5WApSbC4jY812NXdcRkWj4
    - /ip4/192.168.1.139/tcp/9096/ipfs/12D3KooWCWXtr2t7uLRdzwX9CitfLi5WApSbC4jY812NXdcRkWj4
  > IPFS: QmP1aP6pa4N82Yy2NUFzRW7AWSF8St3x3H1b6vd2fg5SPR
    - /ip4/127.0.0.1/tcp/4001/ipfs/QmP1aP6pa4N82Yy2NUFzRW7AWSF8St3x3H1b6vd2fg5SPR
    - /ip4/135.23.195.89/tcp/62370/ipfs/QmP1aP6pa4N82Yy2NUFzRW7AWSF8St3x3H1b6vd2fg5SPR
    - /ip4/192.168.1.139/tcp/4001/ipfs/QmP1aP6pa4N82Yy2NUFzRW7AWSF8St3x3H1b6vd2fg5SPR
    - /ip6/::1/tcp/4001/ipfs/QmP1aP6pa4N82Yy2NUFzRW7AWSF8St3x3H1b6vd2fg5SPR

12D3KooWEr3PcBU3aqCqzDEHADg1CmSZ2CeYyuFCVaCi2DXdhEVX | nato-elementary | Sees 3 other peers
  > Addresses:
    - /ip4/127.0.0.1/tcp/9096/ipfs/12D3KooWEr3PcBU3aqCqzDEHADg1CmSZ2CeYyuFCVaCi2DXdhEVX
    - /ip4/135.23.195.89/tcp/55101/ipfs/12D3KooWEr3PcBU3aqCqzDEHADg1CmSZ2CeYyuFCVaCi2DXdhEVX
    - /ip4/192.168.1.102/tcp/9096/ipfs/12D3KooWEr3PcBU3aqCqzDEHADg1CmSZ2CeYyuFCVaCi2DXdhEVX
    - /ip4/192.168.1.142/tcp/9096/ipfs/12D3KooWEr3PcBU3aqCqzDEHADg1CmSZ2CeYyuFCVaCi2DXdhEVX
  > IPFS: QmcEN7ka3MfvkVhwZu1RGSoPstx72vC5YoZ9Yg4VBXYEq7
    - /ip4/127.0.0.1/tcp/4001/ipfs/QmcEN7ka3MfvkVhwZu1RGSoPstx72vC5YoZ9Yg4VBXYEq7
    - /ip4/135.23.195.89/tcp/63691/ipfs/QmcEN7ka3MfvkVhwZu1RGSoPstx72vC5YoZ9Yg4VBXYEq7
    - /ip4/192.168.1.102/tcp/4001/ipfs/QmcEN7ka3MfvkVhwZu1RGSoPstx72vC5YoZ9Yg4VBXYEq7
    - /ip4/192.168.1.142/tcp/4001/ipfs/QmcEN7ka3MfvkVhwZu1RGSoPstx72vC5YoZ9Yg4VBXYEq7
    - /ip6/::1/tcp/4001/ipfs/QmcEN7ka3MfvkVhwZu1RGSoPstx72vC5YoZ9Yg4VBXYEq7

12D3KooWAG38EVPM1Mrw1YBDe1wuFKS23LiZXq8FGjdypmh6RbgY | lineageos-on-ipfs | Sees 3 other peers
  > Addresses:
    - /ip4/10.20.0.5/tcp/9096/ipfs/12D3KooWAG38EVPM1Mrw1YBDe1wuFKS23LiZXq8FGjdypmh6RbgY
    - /ip4/127.0.0.1/tcp/9096/ipfs/12D3KooWAG38EVPM1Mrw1YBDe1wuFKS23LiZXq8FGjdypmh6RbgY
    - /ip4/159.89.116.13/tcp/9096/ipfs/12D3KooWAG38EVPM1Mrw1YBDe1wuFKS23LiZXq8FGjdypmh6RbgY
  > IPFS: QmSqLAXiJiteNbuNPY4Y5Lp4iKiUmqhCkBZSedZEutktVs
    - /ip4/127.0.0.1/tcp/4001/ipfs/QmSqLAXiJiteNbuNPY4Y5Lp4iKiUmqhCkBZSedZEutktVs
    - /ip4/159.89.116.13/tcp/4001/ipfs/QmSqLAXiJiteNbuNPY4Y5Lp4iKiUmqhCkBZSedZEutktVs
    - /ip6/2604:a880:cad:d0::17:2001/tcp/4001/ipfs/QmSqLAXiJiteNbuNPY4Y5Lp4iKiUmqhCkBZSedZEutktVs
    - /ip6/::1/tcp/4001/ipfs/QmSqLAXiJiteNbuNPY4Y5Lp4iKiUmqhCkBZSedZEutktVs

12D3KooWEcEmcMy2MMHHCqH2w4Hx3TPrpHSKpSWLXKiW5uVKpC74 | nato-hu | Sees 3 other peers
  > Addresses:
    - /ip4/127.0.0.1/tcp/9096/ipfs/12D3KooWEcEmcMy2MMHHCqH2w4Hx3TPrpHSKpSWLXKiW5uVKpC74
    - /ip4/135.23.195.89/tcp/50030/ipfs/12D3KooWEcEmcMy2MMHHCqH2w4Hx3TPrpHSKpSWLXKiW5uVKpC74
    - /ip4/192.168.1.122/tcp/9096/ipfs/12D3KooWEcEmcMy2MMHHCqH2w4Hx3TPrpHSKpSWLXKiW5uVKpC74
    - /ip4/192.168.122.1/tcp/9096/ipfs/12D3KooWEcEmcMy2MMHHCqH2w4Hx3TPrpHSKpSWLXKiW5uVKpC74
  > IPFS: QmUJKWDGYwNsihCrGCxQYEbM2MME28LsMhgCdtfRJ7S8t9
    - /ip4/127.0.0.1/tcp/4001/ipfs/QmUJKWDGYwNsihCrGCxQYEbM2MME28LsMhgCdtfRJ7S8t9
    - /ip4/135.23.195.89/tcp/65074/ipfs/QmUJKWDGYwNsihCrGCxQYEbM2MME28LsMhgCdtfRJ7S8t9
    - /ip4/192.168.1.122/tcp/4001/ipfs/QmUJKWDGYwNsihCrGCxQYEbM2MME28LsMhgCdtfRJ7S8t9
    - /ip4/192.168.122.1/tcp/4001/ipfs/QmUJKWDGYwNsihCrGCxQYEbM2MME28LsMhgCdtfRJ7S8t9
    - /ip6/::1/tcp/4001/ipfs/QmUJKWDGYwNsihCrGCxQYEbM2MME28LsMhgCdtfRJ7S8t9

It doesn't appear in lineageos-on-ipfs's status, but it does appear when checking the status of a single item.

QmWA4Lqv1AQjEbuurrXnCybag1M1jeQbTXUcBkBFoUoG1B :
    > lineageos-on-ipfs : REMOTE: Post http://127.0.0.1:5001/api/v0/pin/ls?arg=QmWA4Lqv1AQjEbuurrXnCybag1M1jeQbTXUcBkBFoUoG1B&type=recursive: dial tcp 127.0.0.1:5001: connect: connection refused | 2019-05-20T22:40:30.190533394-04:00
    > hp-server       : PIN_ERROR: Post http://127.0.0.1:5001/api/v0/pin/ls?arg=QmWA4Lqv1AQjEbuurrXnCybag1M1jeQbTXUcBkBFoUoG1B&type=recursive: dial tcp 127.0.0.1:5001: connect: connection refused | 2019-05-21T03:41:39.245751714Z
    > nato-hu         : REMOTE | 2019-05-21T03:39:14.336839625Z
    > nato-elementary : UNPINNED | 2019-05-21T03:45:07.334229478Z

Logs

23:49:36.972  INFO    service: Initializing. For verbose output run with "-l debug". Please wait... daemon.go:51
23:49:37.022  INFO    cluster: IPFS Cluster v0.10.1+git30ba6f82dd8050e4df302f162ac29b3a4f4f8715 listening on:
        /ip4/127.0.0.1/tcp/9096/ipfs/12D3KooWEr3PcBU3aqCqzDEHADg1CmSZ2CeYyuFCVaCi2DXdhEVX
        /ip4/192.168.1.142/tcp/9096/ipfs/12D3KooWEr3PcBU3aqCqzDEHADg1CmSZ2CeYyuFCVaCi2DXdhEVX
        /ip4/192.168.1.102/tcp/9096/ipfs/12D3KooWEr3PcBU3aqCqzDEHADg1CmSZ2CeYyuFCVaCi2DXdhEVX

 cluster.go:117
23:49:37.022  INFO    restapi: REST API (HTTP): /ip4/127.0.0.1/tcp/9094 restapi.go:456
23:49:37.022  INFO  ipfsproxy: IPFS Proxy: /ip4/127.0.0.1/tcp/9095 -> /ip4/127.0.0.1/tcp/5001 ipfsproxy.go:278
23:49:37.022  INFO    service: Bootstrapping to /ip4/159.89.116.13/tcp/9096/ipfs/12D3KooWAG38EVPM1Mrw1YBDe1wuFKS23LiZXq8FGjdypmh6RbgY daemon.go:202
23:49:37.024  INFO    restapi: REST API (libp2p-http): ENABLED. Listening on:
        /ip4/127.0.0.1/tcp/9096/ipfs/12D3KooWEr3PcBU3aqCqzDEHADg1CmSZ2CeYyuFCVaCi2DXdhEVX
        /ip4/192.168.1.142/tcp/9096/ipfs/12D3KooWEr3PcBU3aqCqzDEHADg1CmSZ2CeYyuFCVaCi2DXdhEVX
        /ip4/192.168.1.102/tcp/9096/ipfs/12D3KooWEr3PcBU3aqCqzDEHADg1CmSZ2CeYyuFCVaCi2DXdhEVX

 restapi.go:473
23:49:37.025  INFO       crdt: crdt Datastore created. Number of heads: 0. Current max-height: 0 crdt.go:213
23:49:37.026  INFO    cluster: Cluster Peers (without including ourselves): cluster.go:478
23:49:37.026  INFO    cluster:     - No other peers cluster.go:480
23:49:37.026  INFO    cluster: ** IPFS Cluster is READY ** cluster.go:493
23:49:37.299  INFO    cluster: 12D3KooWEr3PcBU3aqCqzDEHADg1CmSZ2CeYyuFCVaCi2DXdhEVX: joined 12D3KooWAG38EVPM1Mrw1YBDe1wuFKS23LiZXq8FGjdypmh6RbgY's cluster cluster.go:804

NatoBoram · 2019-05-21T05:35:07Z

Can probably be tested by having peers join the collaborative pinset.

cluster.secret : c89099b9149e2482df0b62f5f77c5db5e04d09dc919d33a60e628e10ed331b2e

consensus.crdt.trusted_peers : 12D3KooWAG38EVPM1Mrw1YBDe1wuFKS23LiZXq8FGjdypmh6RbgY

Bootstrap : ipfs-cluster-service daemon --upgrade --bootstrap /ip4/159.89.116.13/tcp/9096/ipfs/12D3KooWAG38EVPM1Mrw1YBDe1wuFKS23LiZXq8FGjdypmh6RbgY --consensus crdt

hsanjuan · 2019-05-21T07:38:44Z

is cosnensus.crdt.cluster_name set to "ipfs-cluster"? While I can connect, it seems the connection drops rather quickly and I don't think I'm getting the root head of the DAG that would allow me to fetch the state.

NatoBoram · 2019-05-23T04:28:24Z

Ooh. Well, that resolved the issue. So cosnensus.crdt.cluster_name kind of acts like a read permission?

hsanjuan · 2019-05-23T11:47:37Z

@NatoBoram it's the pubsub topic that peers subscribe to for updates, in case you have several clusters running on the same libp2p swarm (i.e. you could run cluster now on the public ipfs network if you really wanted, but using the global DHT likely makes things slower).

hsanjuan · 2019-05-23T22:06:53Z

I'll close this as it seems there's no issue after all, but I'll re-open if anything similar comes up during my testing.

NatoBoram · 2019-05-24T01:52:24Z

i.e. you could run cluster now on the public ipfs network if you really wanted, but using the global DHT likely makes things slower

Sounds fun, I want to test that! How do I do it? What does it do differently?

hsanjuan · 2019-05-24T08:35:48Z

How do I do it?

Runing cluster peers (in crdt mode) with an empty secret should allow them to join the global DHT. With the stuff in #792 you should be able to put IPFS bootstrappers (from ./ipfs/config) in the cluster peerstore file. And then your peers should bootstrap to those and might be able to find each others. I haven't tried any of this.

What does it do differently?

Doesn't do anything differently, just removes the encryption layer that isolated the swarm from other swarms (like the IPFS one).

NatoBoram added kind/bug A bug in existing code (including security flaws) need/review Needs a review labels May 20, 2019

hsanjuan added Unconfirmed and removed need/review Needs a review labels May 20, 2019

hsanjuan added status/invalid Invalid issue and removed Unconfirmed labels May 23, 2019

hsanjuan closed this as completed May 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New peers in a crdt cluster don't receive state #784

New peers in a crdt cluster don't receive state #784

NatoBoram commented May 20, 2019 •

edited

Loading

hsanjuan commented May 20, 2019 •

edited

Loading

NatoBoram commented May 20, 2019

NatoBoram commented May 20, 2019

hsanjuan commented May 20, 2019

NatoBoram commented May 21, 2019 •

edited

Loading

NatoBoram commented May 21, 2019

hsanjuan commented May 21, 2019

NatoBoram commented May 23, 2019

hsanjuan commented May 23, 2019

hsanjuan commented May 23, 2019

NatoBoram commented May 24, 2019

hsanjuan commented May 24, 2019

New peers in a crdt cluster don't receive state #784

New peers in a crdt cluster don't receive state #784

Comments

NatoBoram commented May 20, 2019 • edited Loading

hsanjuan commented May 20, 2019 • edited Loading

NatoBoram commented May 20, 2019

NatoBoram commented May 20, 2019

hsanjuan commented May 20, 2019

NatoBoram commented May 21, 2019 • edited Loading

System

IPFS

IPFS-Cluster

NatoBoram commented May 21, 2019

hsanjuan commented May 21, 2019

NatoBoram commented May 23, 2019

hsanjuan commented May 23, 2019

hsanjuan commented May 23, 2019

NatoBoram commented May 24, 2019

hsanjuan commented May 24, 2019

NatoBoram commented May 20, 2019 •

edited

Loading

hsanjuan commented May 20, 2019 •

edited

Loading

NatoBoram commented May 21, 2019 •

edited

Loading