RA2 Ch4 - Host OS, Kubernetes, Container Runtime and Storage content

[tomkivlin]

Fixes #642

[Levovar]

CentOS / RHEL 8?

[tomkivlin]

At present, RHEL8 isn't compatible with kubeadm. As we progress we can add more based on other LCM tools, but I've started with just kubeadm as the "native" tool from the Kubernetes project. i.e. I'd rather get this published as-is and add more post-Snezka. Is that ok?

[Levovar]

well, K8s also doesn't officially support e.g. multi-tenancy, Docker versions after 17. something etc., but still we state requirements :)
but I understand your reasoning, no strong feelings one way or the other
however if we state CentOS/RHEL 7 is okay, then we need discuss kernel versions too IMO

[pgoyal01]

"The distribution and version of Kubernetes used must be listed in the Kubernetes Distributions and Platforms document which lists all those that are Certified as being conformant with the CNCF Kubernetes specification."

Suggest changes. Possibly:
CNTT will use one of the Kubernetes distributions listed in the Kubernetes Distributions and Platforms document; the document lists distributions certified as being conformant with the CNCF Kubernetes specification.

[tomkivlin]

How about:

In alignment with the Kubernetes version support policy, a Reference Implementation must use one of three latest minor versions (n-2) - e.g. if the latest version is 1.17 then the RI must use either 1.17, 1.16 or 1.15. The Kubernetes distribution or product that is used in the RI must be listed in the Kubernetes Distributions and Platforms document and marked (X) as conformant for the Kubernetes version that is being used.

[pgoyal01]

@tomkivlin That is great.

[pgoyal01]

"At least one worker node per availability zone or fault domain to ensure the high availability and resilience of workloads managed by Kubernetes"

As stated it seems to imply that we only need at least one worker node in each AZ/fault domain for workload resiliency. This needs rewording for workload resiliency: "Workload resiliency requires the need to run them on at least one worker node per availability zone or fault domain."

[tomkivlin]

As stated it seems to imply that we only need at least one worker node in each AZ/fault domain for workload resiliency.

That is correct, and I'm not sure how your suggestion differs from the existing wording? If I was to move "Consist of" from the first bullet to the statement before the list, would that make more sense?

[pgoyal01]

@tomkivlin Maybe the sentence needs improving but I tried to state that the workload must be "run" on worker nodes in each AZ: "Workload ... need to run them on at least one worker node per availability zone" -- if not clear then does the following reword help: "Workload resiliency requires the need to run the workload on at least one worker node per availability zone or fault domain."

[tomkivlin]

Ah I see - I think that is outside the scope of this RA. The requirement is to support workload resilience (which having one worker node in each AZ does), not to specify how/where the workloads are run. Bit nit-picky I know, but I am keen to avoid too much specification that isn't about the platform. Is that ok?

[TamasZsiros]

Good text overall. I have a bit of an issued with subchapter 4.2:
I understand Host OS-es specifiedare the ones compatible with kubeadm, however Peter and I will present at Prague a proposal on OS modularity. That proposal - if we can gain general consensus around it - would invalidate 4.2 as it is written today (or at least the list of Host OS-es supported would have to be modified - since the criteria of having a Linux kernel of a certain version would have to be introduced).
My suggestion is to keep this PR open and decide what to do with 4.2 after (or at) the meeting in Prague.

Mmm, good point - I am 100% supportive of that modularity concept too. As per @Levovar 's comments as well, what I'll do is modify that and have a skeleton table, so we can merge the PR (otherwise, we miss the date for Snezka release) and create backlog item to populate - does that sound ok?

-- Yes, ok!

[TamasZsiros]

@TamasZsiros do the changes in f4f7c71 address your concern?

cc @Levovar as well, given previous comments.

Yes it does, thank you.

[Levovar]

@TamasZsiros do the changes in f4f7c71 address your concern?

cc @Levovar as well, given previous comments.

hmm, unfortunately no :)
the reason I said "we must elaborate on kernel versions if CentOS/RHEL7 is supported" because their stock kernel is not suitable for a production CaaS.
so, if these OSs are supported, their kernels need to be changed, and IMO we should make it very explicit somewhere in the RA2
the content behind the link does not state anything related to kernel, but kernel features we reference throughout the RA2 definitely have dependencies: IPVLAN, ovelayfs2 (with project quotas) on XFS / ext4, CFS quotas etc. all require sthing sthing sthig 4.X
4.14, at the very least. 4.18 would be better. 4.19 would be the best for a "minimum" requirement as this version contains the CFS quota corrections too

[tomkivlin]

the reason I said "we must elaborate on kernel versions if CentOS/RHEL7 is supported" because their stock kernel is not suitable for a production CaaS.

Can you reference some evidence for that - for example OpenShift uses RHEL7, are you suggesting it's not suitable for production use? There will be many customers concerned I'm sure :)

the content behind the link does not state anything related to kernel

Sorry, there is a later commit where I added the correct link, which contains the following:

Kubernetes system requirements:
if running on linux:
[error] if not Kernel 3.10+ or 4+ with specific KernelSpec

I have no problem restricting further in future PRs, but we need to be complete about it - i.e. reference the system requirement from Kubernetes documentation that states it is required, and trace back to a requirement in chapter 2. e.g. which requirement is CFS needed for, and which Kubernetes doc states it is required?

I would suggest that we add as a backlog item(s) to modify in later PRs.

[Levovar]

Can you reference some evidence for that - for example OpenShift uses RHEL7, are you suggesting it's not suitable for production use? There will be many customers concerned I'm sure :)

hmm. kernel release notes? :)
if you can live with the restrictions imposed upon you by an older kernel, sure it is okay. I'm not here to bash or praise products or solutions, but facts are facts: some features we value highly need newer kernels to even work, or just to work properly
for more info on RH I suggest asking RH representatives how do they support e.g. IPVLAN in RHEL7 :)

Anyway some basic references related to the few examples I have used:
https://cateee.net/lkddb/web-lkddb/IPVLAN.html
kubernetes/kubernetes#67577
https://docs.docker.com/storage/storagedriver/overlayfs-driver/ (this one was at least backported to RHEL stock kernel it seems, but nevertheless there were many bug corrections later AFAIK)
https://lwn.net/Articles/671627/

I have no problem restricting further in future PRs, but we need to be complete about it - i.e. reference the system requirement from Kubernetes documentation that states it is required, and trace back to a requirement in chapter 2. e.g. which requirement is CFS needed for, and which Kubernetes doc states it is required?

K8s CPU management itself is based CFS quotas. E.g. when you set a "CPU limit", you actually set a CFS quota.
Inspect above link - you might want it to work properly, without additional throttling in your K8s environment :)

I would suggest that we add as a backlog item(s) to modify in later PRs.

I'm fine however you want to handle it, but if it is stated in one document that "you shall use this & that feature and CNI and plugin", and then it is stated in another that "an operating system flat-out not supporting some of the features, or only supporting them with known bugs is OK" -> that's a direct contradiction to me

[Levovar]

BTW my two cents, generally speaking: it is okay to use the upstream K8s project requirements as a starting point, however it is important to note Ra2 does not define a K8s deployment.
It defines a TelCo CaaS based on K8s, that is K8s plus all the other stuff we have decided earlier to add to the scope of a "CaaS".

Therefore requirements directly coming from the K8s project are essential, but not satisfactory in themselves. Our end requirements and restrictions must account for the non-K8s projects too, therefore probably end up much stricter than the base K8s criteria

[tomkivlin]

BTW my two cents, generally speaking: it is okay to use the upstream K8s project requirements as a starting point, however it is important to note Ra2 does not define a K8s deployment.
It defines a TelCo CaaS based on K8s, that is K8s plus all the other stuff we have decided earlier to add to the scope of a "CaaS".

Therefore requirements directly coming from the K8s project are essential, but not satisfactory in themselves. Our end requirements and restrictions must account for the non-K8s projects too, therefore probably end up much stricter than the base K8s criteria

Totally agree - but those extra restrictions must be driven by and traceable back to requirements in chapter 2.

[tomkivlin]

I'm fine however you want to handle it, but if it is stated in one document that "you shall use this & that feature and CNI and plugin", and then it is stated in another that "an operating system flat-out not supporting some of the features, or only supporting them with known bugs is OK" -> that's a direct contradiction to me

It's a fair point. I have created Issue #920 to capture this for a future PR.

[tomkivlin]

@TamasZsiros can you approve please (still marked as you requested changes)?

[tomkivlin]

In order to merge for Snezka, I can see in the comments that Tamas is ok with the changes, it's just the approval response that needs to change. As a result it is ready to merge @rabiabdel .