Security Focused : Architectural Decision For mTLS Feature .

[sagarkhandagre998]

Topic - Why are we introducing two separate gRPC servers for mTLS ?

Teams Question -

Aryan -
1. Why are we introducing two separate gRPC servers?
My understanding is that mTLS will be enabled on both servers by default. I understand that TLS for the LogServer can be made optional via a flag and Management will always be enabled, but we could go ahead with mTLS enable by default and leave it on the client to whether disable it?  We can look into making use og gRPC interceptors for additional authentication and authorization if required, while keeping the overall architecture simpler.

2. HealthCheck service
We recently moved the HealthCheck service to a separate port because Kubernetes liveness probes do not support TLS. With this proposal, we would  now end up maintaining three separate gRPC ports.

Follow Up By me -

Hey Aryan , Thanks for taking time to review the design and discussed with the team. I also spent some time thinking about these points as well. My reasoning for proposing two servers isn't primarily because of mTLS itself, but because of the separation of responsibilities between the APIs. (Followed by speration of concern design principle).

1. Why introduce two gRPC servers?

I agree that if the only objective were "enable mTLS by default", then a single gRPC server would be sufficient. Both approaches can achieve transport security.

The motivation behind the split is architectural rather than just implementing mTLS .

As we know the services currently exposed by KubeArmor naturally fall into two different categories:

1. Data plane / Observability

• LogService (WatchLogs / WatchAlerts)

2. Control plane / Management

• PolicyService
• ProbeService
• StateAgent
• Reflection

This two groups has diff characteristics

• different trust requirements
• diff authentication requirements
• diff exposure requirements
• diff authorization requirements in the future

This follows a common cloud-native design principle:
Control Plane ≠ Data Plane

Many infrastructure projects intentionally separate these interfaces instead of relying solely on application-layer authorization.

The goal is not simply stronger authentication but creating a clear security boundary around privileged operations.
with two endpoints:

• the LogServer remains a dedicated observability interface,
• the ManagementServer becomes the only entry point for privileged operations.

This also means future enhancements such as RBAC, SPIFFE identities, authorization interceptors, or stronger administrative authentication remain isolated to the management plane without affecting the logging path.

From a maintenance perspective, the server registration also becomes simpler since each server owns a single responsibility rather than mixing observability and management services behind one listener.

Current Pr (changes made after i shared docs ) - I have made it configurable , both log server and management sever can be operate by flag.

Overall my thinking was to optimized for Long Term Architecture rather than just the immediate mTLS requirement .Even if both servers use mTLS today , I believe seperating the observability plane from the management plane provides a cleaner security model, aligns with the pattern used in the cloud-native infrastructure projects , and gives us more flexibility as the  management API evolves . Interceptors can still be layered on the top of management server later ( that was already plan ) for finer-grained authentication and authorization without changing that architectural boundary.

[sagarkhandagre998]

@achrefbensaad @Aryan-sharma11 @AryanBakliwal @ravkishu @murtaza-u

[achrefbensaad]

imo, as long as we wont be issuing separate mtls certs for the new management endpoint, the separation has no real advantage. yes it. makes refactoring easier in the future but by how much ? few lines of code wont change the scoping of work.
if we want to go with two separate endpoints, we need the following:

• separate mtls certificates for log and mgmt endpoint
• update the feeder logic to lookup and use the log endpoint certificates
• update karmor to select the appropriate certificate when making the grpc call.

[sagarkhandagre998]

I Agree with your point , if we want to justify two separate endpoints from a security perspective, we should also separate the mTLS material for log and management.

A good target design would be :

Add separate certs/trust domains for both:
- log endpoint certs
- management endpoint certs

Then:

-LogServer loads log server certs and trusts log client CA.
-ManagementServer loads management server certs and trusts management client CA.
-karmor log commands use log client certs.
-karmor policy/probe/state commands use management client certs.
-Tests/CI generate and consume both cert sets.

I would appreciate the team's decision for mTLS implementation so that we can continue.

[sagarkhandagre998]

Hi @achrefbensaad ,

I checked the current cert loading behavior, and there is one related point we should clarify before implementing mTLS.
Right now the self provider does not bootstrap a CA by itself. It expects ca.crt and ca.key to already be present under tlsCertPath, and then it dynamically generates the server certificate from that CA.

So if we move to separate mTLS material for log and management, we need to decide how those CA files are expected to exist.

For Kubernetes/operator mode, it looks like the operator already creates and mounts the KubeArmor CA secret, so the current self provider works there. But i think for non k8s production, I am not fully clear on the expected flow.

Should we assume production users/operators will pre-provision the CA material for both endpoints, for example:

/var/lib/kubearmor/tls/log/ca.crt
/var/lib/kubearmor/tls/log/ca.key
/var/lib/kubearmor/tls/management/ca.crt
/var/lib/kubearmor/tls/management/ca.key

Or should KubeArmor/operator be responsible for creating separate CA material for log and management?

I wanted to confirm the expected CA bootstrap model?

[sagarkhandagre998]

@Aryan-sharma11 I would appreciate the team's decision on this implementation ?

[Aryan-sharma11]

Hi @sagar khandagre Yes, we can go ahead with the implementation, there few things to be considered. Regarding you question about should kubearmor/operator be responsible for creating separate CA material for log and management? I think we can handle the creation of CA in kubearmor-client for unorchestrated modes. WDYT?

[sagarkhandagre998]

@Aryan-sharma11

I agree that kubearmor-client should handle CA creation for unorchestrated modes. The install/pki.go already has GenerateCA(), so we just need a kubearmor-client vm init subcommand that:

• Generates CA + client cert for both observability and management paths.
• Writes CA files (ca.crt + ca.key) to a tarball for deployment to the target host - KubeArmor's "self" provider reads the CA and generates ephemeral server certs at startup.
• Saves client cert (client.crt + client.key) locally for karmor to use when connecting.
• No server cert needed - KubeArmor generates those ephemerally in-memory from the CA.

For the current PR (helm/KubeArmor changes), the SelfSignedCertLoader still needs to auto-bootstrap CA when files don't exist (for operator/helm mode where the CA secret is mounted). This doesn't conflict - in K8s the CA is already pre-provisioned, so EnsureCA() would be a no-op.

Separate PR against kubearmor-client for vm init, tracked in a follow-up issue. WDYT?

[sagarkhandagre998]

The corrected flow:

Admin's Machine                         Target Host (bare VM)
──────────────                          ─────────────────────
1. karmor vm init
   ├── ca.crt + ca.key ──────────────────► 2. Copied to host
   └── client.crt + client.key      
                                        3. KubeArmor starts
                                        4. Reads ca.crt + ca.key from disk
                                        5. Creates ephemeral server cert in memory
                                        6. Starts mTLS server with that server cert

7. karmor logs
   └── Uses client.crt + ca.crt locally
   ──→ Server sends ephemeral server cert
   ──→ Client verifies it with ca.crt (from step 1) ✓
   ──→ Server verifies client.crt with ca.crt ✓

• No TOFU vulnerability
• CA private key is generated on the admin's machine, not the target host
• Client cert never touches the target host
• kubearmor-client already has the PKI primitives in install/pki.go - just needs a vm init subcommand to wire it up

[sagarkhandagre998]

@Aryan-sharma11

The Final Design Architecture Would be this -

Data Plane - Obervability

• Feeder Object - Responsible for registering Log Server on port : 32767
• Log Server is responsible for registering logService Only.

Listener A: TCP :32767
(configurable) -
Observability TLS cert

Control Plane - Management

• Management Object - Responsible for registering Management Server on port 32768.
• Management Server is responsible for registering 3 Services : policyService, stateAgent, probService.

Listener B: Unix Domain
Socket or localhost-only
TCP - Management
TLS/mTLS cert

[sagarkhandagre998]

@achrefbensaad @Aryan-sharma11

Hi

I was reviewing the GenerateCA() function and noticed an unusual pattern where two separate RSA key pairs are generated:

• GenerateCert(cfg) creates key-1 + template-1.
• GenerateSelfSignedCert(crtTemp, cfg) internally calls GenerateCert again, creating key-2 + template-2.
• The resulting CA cert is signed with key-1 but embeds key-2's public key; key-2 is returned as the CA private key.

This means the signing key and embedded public key don't correspond -Probably the CA cert would fail self-signature verification. I saw the standard practice (Go stdlib, k8s client-go, cfssl) is to generate a single key and call x509.CreateCertificate(rand.Reader, &template, &template, &key.PublicKey, key).

Was there a specific reason this two-key approach was chosen? I wanted to understand the intent.