Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initContainer defaults preventing existing YAML from working #240

Closed
aquarapid opened this issue Jan 25, 2022 · 0 comments · Fixed by #370
Closed

initContainer defaults preventing existing YAML from working #240

aquarapid opened this issue Jan 25, 2022 · 0 comments · Fixed by #370
Labels
bug Something isn't working

Comments

@aquarapid
Copy link
Member

I have a (test) operator cluster YAML that has a keyspace section looking like this:

  keyspaces:
  - name: main
    partitionings:
    - equal:
        parts: 2
        shardTemplate:
          databaseInitScriptSecret:
            name: example-cluster-config
            key: init_db.sql
          replication:
            enforceSemiSync: true
          tabletPools:
          - cell: uscentral1a
            type: replica
            replicas: 1
            vttablet:
              extraFlags:
                db_charset: utf8mb4
              resources:
                requests:
                  cpu: 10m
                  memory: 256Mi
            mysqld:
              resources:
                requests:
                  cpu: 10m
                  memory: 256Mi
            dataVolumeClaimTemplate:
              accessModes: ["ReadWriteOnce"]
              resources:
                requests:
                  storage: 5Gi
              storageClassName: uscentral1a-ssd
          - cell: uscentral1c
            type: replica
            replicas: 1
            vttablet:
              extraFlags:
                db_charset: utf8mb4
              resources:
                requests:
                  cpu: 10m
                  memory: 256Mi
            mysqld:
              resources:
                requests:
                  cpu: 10m
                  memory: 256Mi
            dataVolumeClaimTemplate:
              accessModes: ["ReadWriteOnce"]
              resources:
                requests:
                  storage: 5Gi
              storageClassName: uscentral1c-ssd
          - cell: uscentral1f
            type: replica
            replicas: 1
            vttablet:
              extraFlags:
                db_charset: utf8mb4
              resources:
                requests:
                  cpu: 10m
                  memory: 256Mi
            mysqld:
              resources:
                requests:
                  cpu: 10m
                  memory: 256Mi
            dataVolumeClaimTemplate:
              accessModes: ["ReadWriteOnce"]
              resources:
                requests:
                  storage: 5Gi
              storageClassName: uscentral1f-ssd

This used to work just fine. But since PR #216 I get the multiple errors (one for each vttablet) when applying this (can be seen with kubectl get events or in the operator logs):

2m38s       Warning   CreateFailed             vitessshard/example-main-x-80-84bc4397                                   failed to create Pod example-vttablet-uscentral1f-2191363457-7dead7ff: Pod "example-vttablet-uscentral1f-2191363457-7dead7ff" is invalid: [spec.initContainers[0].resources.requests: Invalid value: "256Mi": must be less than or equal to memory limit, spec.initContainers[1].resources.requests: Invalid value: "256Mi": must be less than or equal to memory limit]

I can resolve this in one of two ways:

  • Reverting Set vttablet init container resource limits #216
  • Reducing the memory requests for the vttablets to 128Mi. However, this isn't a real solution, since it would pack too many vttablet pods into a node where I want to run fewer/larger vttablets...

Environment is GKE (1.19.16-gke.1500).

@aquarapid aquarapid added the bug Something isn't working label Jan 25, 2022
brandt added a commit to brandt/vitess-operator that referenced this issue Mar 6, 2022
Sets the resource requirements on each of the default vttablet init containers to the same request and limit values as the vttablet container itself.

This fixes the issue reported in planetscale#212 where pods would fail to be created on clusters with strict quota requirements because no limit was set on the init containers.

## History

This is a reimplementation of planetscale#216 with the change that it no longer sets a default resource _limit_.

This fixes the issue reported in planetscale#240 and discussed in planetscale#244 where the vttablet pod would fail to be created when the user sets a request larger than the default limit without also setting a larger limit. The user's nil limit would not override the default value because `update.ResourceRequirements(dst, src)` won't update `dst` with a nil `src` ([for good reasons](https://github.com/planetscale/vitess-operator/blob/da7efd4c945193864f5996cb793055449c4de97a/pkg/operator/update/podspec.go#L201-L203)).

With this change, users can once again increase the request by itself. If a cluster requires a limit, the user will already need to have set it on for vttablet so we can use the same value to override the init containers' default nil limit.

fixes planetscale#212
fixes planetscale#240

Signed-off-by: J. Brandt Buckley <brandt@runlevel1.com>
GuptaManan100 pushed a commit that referenced this issue Feb 2, 2023
Sets the resource requirements on each of the default vttablet init containers to the same request and limit values as the vttablet container itself.

This fixes the issue reported in #212 where pods would fail to be created on clusters with strict quota requirements because no limit was set on the init containers.

## History

This is a reimplementation of #216 with the change that it no longer sets a default resource _limit_.

This fixes the issue reported in #240 and discussed in #244 where the vttablet pod would fail to be created when the user sets a request larger than the default limit without also setting a larger limit. The user's nil limit would not override the default value because `update.ResourceRequirements(dst, src)` won't update `dst` with a nil `src` ([for good reasons](https://github.com/planetscale/vitess-operator/blob/da7efd4c945193864f5996cb793055449c4de97a/pkg/operator/update/podspec.go#L201-L203)).

With this change, users can once again increase the request by itself. If a cluster requires a limit, the user will already need to have set it on for vttablet so we can use the same value to override the init containers' default nil limit.

fixes #212
fixes #240

Signed-off-by: J. Brandt Buckley <brandt@runlevel1.com>
GuptaManan100 added a commit that referenced this issue Feb 2, 2023
* Set vttablet init container resource requests and limits

Sets the resource requirements on each of the default vttablet init containers to the same request and limit values as the vttablet container itself.

This fixes the issue reported in #212 where pods would fail to be created on clusters with strict quota requirements because no limit was set on the init containers.

## History

This is a reimplementation of #216 with the change that it no longer sets a default resource _limit_.

This fixes the issue reported in #240 and discussed in #244 where the vttablet pod would fail to be created when the user sets a request larger than the default limit without also setting a larger limit. The user's nil limit would not override the default value because `update.ResourceRequirements(dst, src)` won't update `dst` with a nil `src` ([for good reasons](https://github.com/planetscale/vitess-operator/blob/da7efd4c945193864f5996cb793055449c4de97a/pkg/operator/update/podspec.go#L201-L203)).

With this change, users can once again increase the request by itself. If a cluster requires a limit, the user will already need to have set it on for vttablet so we can use the same value to override the init containers' default nil limit.

fixes #212
fixes #240

Signed-off-by: J. Brandt Buckley <brandt@runlevel1.com>

* refactor: move defaults to the correct location

Signed-off-by: Manan Gupta <manan@planetscale.com>

---------

Signed-off-by: J. Brandt Buckley <brandt@runlevel1.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: J. Brandt Buckley <brandt@runlevel1.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
1 participant