helm, info, status: Check and report reachability sequencer + refactor

[giner]

Possible status values:

• Sequencer: 0 (reachable and not lagging), 1 (lagging), 2 (unreachable), 3 (unreachable and lagging).

[isegall-da]

/cluster_test

[github-actions]

Deploy cluster test triggered for Commit 01a3ced14e736c8d92fc1b7b9fb3d072cb36585b in , please contact a Contributor to approve it in CircleCI: https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/71127

[isegall-da]

/cluster_test

[github-actions]

Deploy cluster test triggered for Commit 01a3ced14e736c8d92fc1b7b9fb3d072cb36585b in , please contact a Contributor to approve it in CircleCI: https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/null

[isegall-da]

/cluster_test

[github-actions]

Deploy cluster test triggered for Commit 01a3ced14e736c8d92fc1b7b9fb3d072cb36585b in , please contact a Contributor to approve it in CircleCI: https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/71265

[martinflorian-da]

Deploy cluster test triggered for Commit 01a3ced14e736c8d92fc1b7b9fb3d072cb36585b in , please contact a Contributor to approve it in CircleCI: https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/71265

green: https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/71266/workflows/49d239f5-71f5-46d3-9c3b-0780a226216e

[martinflorian-da]

I agree with Itai that this is hard to review. The downside of getting something wrong here is limited though IMO, so happy to rush this a bit to make the release cut tomorrow, and we can always fix things later on...

@giner I'll fix your merge conflict and set automerge now, so we get a little bit of more testing before the release cut.

[martinflorian-da]

I actually need to retract my approval following some input from a watchful colleague ™️. Downloading remote binaries in a bash script is fairly sketchy from a supply chain security standpoint; at the very least I'd like us to review this with less haste.

How robust is that sha check against future code changes to this script, for example?

Perhaps the clean way is to use a different image that has grpcurl baked in?

[giner]

I agree with Itai that this is hard to review.

Let me try to refactor it a bit on the next iteration (remove some verbose bash 3 compatibility stuff, split into smaller easier to read, self described pieces). If it's still hard to read after that I'll look into rewriting it Python.

[giner]

Perhaps the clean way is to use a different image that has grpcurl baked in?

I couldn't find a good image containing all: jq, curl, prom2json and grpcurl. If downloading at runtime is no go we'll have to manage another image.

[moritzkiefer-da]

There are two somewhat orthogonal issues for me:

1. Downloading tools at runtime is a hard no for me here.
2. grpcurl itself is in a bit of a dodgy state. If you look through the issue tracker there are a bunch of issues for dependencies with known vulnerabilities without a release that fixes it and it gets flagged by most docker image scanners. That's why we just removed it from the base image of all the actual apps. I'm not necessarily saying that those vulnerabilities do apply the way we use it here but I don't even want to have to reason about this and explain this every time an auditor runs a scan.

I suspect in most SVs setups the info pod has direct access to things like participant admin APIs so it is highly security critical so we definitely should err on the side of being overly cautious.

If we can build an image containing https://github.com/grpc-ecosystem/grpc-health-probe to get the same info, that seems like a reasonable option. I don't really know any other alternative that is well maintained and doesn't get flagged. We could write some custom python program or similar but at that point, I'd also start to somewhat question whether the complexity is worth the ROI.

[giner]

Rebased on main and squashed

[giner]

Rebased on the current main

[isegall-da]

I still think that the readability gain of doing this in python would have greatly outweighed the downsides, but I accept that you see things differently.

Done