Tessera process continue to throw exceptions and private transactions don't go through on distributed node/+ tessera setup

[golashr]

System information
Geth version: 1.8.18-stable
Quorum version:quorum-v2.2.3
Tessera version: quorumengineering/tessera:0.9.1
OS & Version: Linux/OSX
Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-47-generic x86_64)

We have Geth+Tessera nodes deployed on distributed machines. Peer tessera nodes are mentioned in the peer list of tessera-config.json of each tessera node. The issue is, the tessera node discovers peer nodes outside the specified peer network too. If the discovered peer list seems to have an issue with connectivity, it throws exceptions and it continues.

{"log":"06:50:22.272 [pool-1-thread-1] ERROR c.q.t.t.TesseraScheduledExecutor - Error when executing action PartyInfoPoller\n","stream":"stdout","time":"2019-05-09T06:50:22.272856945Z"}
{"log":"06:50:22.272 [pool-1-thread-1] ERROR c.q.t.t.TesseraScheduledExecutor - Error when executing action\n","stream":"stdout","time":"2019-05-09T06:50:22.272869291Z"}
{"log":"javax.ws.rs.ProcessingException: java.net.NoRouteToHostException: Host is unreachable (Host unreachable)\n","stream":"stdout","time":"2019-05-09T06:50:22.272872429Z"}
{"log":"\u0009at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:284) ~[tessera-app.jar:na]\n","stream":"stdout","time":"2019-05-09T06:50:22.272892301Z"}
{"log":"\u0009at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:278) ~[tessera-app.jar:na]\n","stream":"stdout","time":"2019-05-09T06:50:22.272895579Z"}
{"log":"\u0009at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:753) ~[tessera-app.jar:na]\n","stream":"stdout","time":"2019-05-09T06:50:22.272897997Z"}
{"log":"\u0009at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[tessera-app.jar:na]\n","stream":"stdout","time":"2019-05-09T06:50:22.272900301Z"}
{"log":"\u0009at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[tessera-app.jar:na]\n","stream":"stdout","time":"2019-05-09T06:50:22.272902551Z"}
```
Tessera should have mechanism to filter out unreachable nodes. 

Subsequently, we can not execute any private transactions on such tessera nodes.

[prd-fox]

Hey.

Two things here:

1. I agree that the logging can be a bit verbose with disconnected peers, especially since you have intentionally shut those nodes down for good.

2. Can you elaborate on tessera discovers peer nodes outside the specified peer network? Is this a fully controlled network? I am curious to know how unknown nodes got into the network list. You may be interested in this: disabling peer discovery

[golashr]

Thanks, @prd-fox. Well, we are building an open blockchain network with tesseras so we have to keep the discovery ON. So we have multiple such networks running and node of one network seem to discover nodes of other networks, as I described above.

In one of our blockchain setup, we have tessera nodes running with Docker containers which are obviously given internal bridge IP address i.e. 172.. P2P serverIPAddress uses 172. and it passes the same to other nodes via /partyinfo

[prd-fox]

Okay, so discovery must remain on, presumably because you may be adding other nodes to that network in the future?

You have multiple networks, but they must not cross over. I am imagining a scenario like this:

Network1:
172.0.0.1:9000
172.0.0.1:9001
172.1.0.1:8000

Network2:
172.0.0.1:28000
172.1.0.1:28001
172.2.0.1:28002

Now the problem is that the two networks have somehow crossed over to create one single network with all 6 nodes?

[golashr]

Yes, the network will look like this.
Let me give another scenario which is not cross blockchain network but on the same network.
Let's say, there are four 172 nodes setup on the same machine. They are able to write private transactions to each other. Further, I add one more node to the existing setup from remote machine e.g. 121.* Now, the new node will have 172* peer node in its peer list thru /partyinfo call. And this exception will start coming.

[prd-fox]

It makes sense.
The 121.* node doesn't have anyway to communicate with the 172.* nodes, even though they are part of the same network.

This is a misconfiguration, since the Docker containers should be set up with an externally accessible IP address, but I agree that the extra log noise is annoying. We will look to reduce that.

Additionally, in the scenario I gave above, the only way for the networks to be merged is for one of the nodes to have added a node from the other network to it's startup list (or at runtime using the specific API endpoint for this). Again, the log noise will be annoying, but they two networks won't automatically merge, someone configured their node to do this.

[golashr]

@prd-fox I submitted a pull request for this ticket. You can review and accept. Of course, we need to make one change, which is more universally accepted field name than "ledgerID" which is used in the code at present.

[prd-fox]

Hi @golashr,

We discussed the PR within ourselves and came to the conclusion that we are specifically targeting a permissioned environment rather than a public, non-permissioned one. Since in a permissioned environment, all the nodes should be happy to connect to each other, with other settings (such as the peer discovery flag, TLS certs) used to keep unwanted participants out.

I have created another issue (#805) where we have identified a number of places that our logging can be improved, so you can track that there.

Thanks for the contribution, we appreciate the interest in Tessera (and Quorum as a whole) even if we didn't accept the PR

[golashr]

@prd-fox I am not sure how logging or setting it up with TLS certs will help the cause. Peer discovery can't be off as I explained before. TLS will only secure the network traffic but tessera node will still connect with externally accessible IP addresses. Appreciate if you can keep the issue open for further discussion. Thanks.

[prd-fox]

I'm still thinking in terms of permissioned networks, which yeah I guess you're not using here.

[prd-fox]

After reviewing this again with the team, we came to the decision to still not merge this.
The change makes sense for your use-case (non-permissioned networks), but that isn't the use case we are targeting.
I can empathise not wanting to maintain a second codebase yourself, but we don't want to add anything that may cause connectivity problems + debugging time for something that we don't intend to target.

[Krish1979]

@golashr - Do you have any more comments/questions before we close this issue ?