Proposal: Add Discriminators in All Unions/OneOf APIs

[mengqiy]

Overview

We have a number of cases in the API where only one of a set of fields is allowed to be specified, aka undiscriminated union / oneof.

VolumeSource is the canonical example: it has fields such as emptyDir, gcePersistentDisk, awsElasticBlockStore and other 20 fields. Only one of these fields can be specified.

We should add discriminators to union / oneof APIs, since it has several advantages.

Original issue is described in kubernetes/kubernetes#35345

Advantages

Adding discriminators to all unions/oneof cases would have multiple advantages:

1. Clients could effectively implement a switch instead of if-else trees to inspect the resource -- look at discriminator and lookup the corresponding field in a map (though differences in capitalization of the first letter in the API convention currently prevents the discriminator value from exactly matching the field name).

2. The API server could automatically clear non-selected fields, which would be convenient for kubectl apply and other cases.

Analysis

List of Impacted APIs

In pkg/api/v1/types.go:

• VolumeSource
• PersistentVolumeSource
• Handler
• ContainerState
• PodSignature: It has only one field, but the comment says "Exactly one field should be set". Maybe we will add more in the future?

In pkg/authorization/types.go:

• SubjectAccessReviewSpecExactly one of ResourceAttributes and NonResourceAttributes must be set. But other non-union fields also exist in this API.
• SelfSubjectAccessReviewSpec

In pkg/apis/extensions/v1beta1/types.go:

• DeploymentStrategy
• NetworkPolicyPeer
• IngressRuleValue: It says "exactly one of the following must be set". But it has only one field.

Behavior

If the discriminator were set, we'd require that the field corresponding to its value were set and the APIServer (registry) could automatically clear the other fields.

If the discriminator were unset, behavior would be as before -- exactly one of the fields in the union/oneof would be required to be set and the operation would otherwise fail validation.

We should set discriminators by default. This means we need to change it accordingly when the corresponding union/oneof fields were set and unset. If so, clients can rely on this for purpose (1).

Proposed Changes

API

Add a discriminator field in all unions/oneof APIs.

The discriminator should be optional for backward compatibility. There is an example below, the field Type works as a discriminator.

type PersistentVolumeSource struct {
	// +optional
	GCEPersistentDisk *GCEPersistentDiskVolumeSource `json:"gcePersistentDisk,omitempty" protobuf:"bytes,1,opt,name=gcePersistentDisk"`
	// +optional
	AWSElasticBlockStore *AWSElasticBlockStoreVolumeSource `json:"awsElasticBlockStore,omitempty" protobuf:"bytes,2,opt,name=awsElasticBlockStore"`
	
...

	// Discriminator for PersistentVolumeSource, it can be "gcePersistentDisk", "awsElasticBlockStore" and etc.
	// +optional
	Type *string `json:"type,omitempty" protobuf:"bytes,24,opt,name=type"`
}

API Server

We need to add defaulting logic described in the Behavior section.

kubectl

No change required on kubectl.

Example Discriminators

Discriminator are Set by Default

Assume we have added a field as discussed in section API changes

We first use kubectl apply -f create a PersistentVolume using the following config:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
  annotations:
    volume.beta.kubernetes.io/storage-class: "slow"
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  hostPath:
    path: /tmp

The subsequent kubectl get should have the discriminator field set by API server:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-class: slow
  creationTimestamp: 2016-12-22T00:56:31Z
  name: pv0001
  namespace: ""
  resourceVersion: "1059564"
  selfLink: /api/v1/persistentvolumespv0001
  uid: 7a57e42b-c7e1-11e6-aa89-42010a800002
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  hostPath:
    path: /tmp
  persistentVolumeReclaimPolicy: Recycle
  # Discriminator showing the type is "hostPath"
  type: hostPath
status:
  phase: Available

Automatically Clear Unselected Fields

Issue kubernetes/kubernetes#34292 will be fixed if spec.strategy.type is treated as a discriminator.

Create a deployment using kubectl apply-f

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

Get the deployment back. Fields spec.strategy.type and spec.strategy.rollingUpdate have been defaulted.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"kind":"Deployment","apiVersion":"extensions/v1beta1","metadata":{"name":"nginx-deployment","creationTimestamp":null},"spec":{"replicas":1,"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"nginx"}},"spec":{"containers":[{"name":"nginx","image":"nginx","ports":[{"containerPort":80}],"resources":{}}]}},"strategy":{}},"status":{}}
  creationTimestamp: 2016-12-22T01:23:34Z
  generation: 1
  labels:
    app: nginx
  name: nginx-deployment
  namespace: default
  resourceVersion: "1062013"
  selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/nginx-deployment
  uid: 416ef165-c7e5-11e6-aa89-42010a800002
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  strategy:
    # Defaulted by API server
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    # Defaulted by API server
    type: RollingUpdate
...

Then we update the config file by adding

  strategy:
    type: Recreate

Apply the new config by kubectl apply -f.

The operation should succeed now, because now the API server knows to clear field spec.strategy.rollingUpdate after updating spec.strategy.type.

@kubernetes/sig-api-machinery-misc @kubernetes/api-reviewers @kubernetes/kubectl

[thockin]

While I accept that this change is desired, and I don't have a problem with it, I have some comments (shocking, I know!). First, I find the arguments for this to be weak. Switch vs if/else is relatively insignificant from code or perf, and as pointed out the constants and the fields end up with different names anyway. Second, I don't know that clearing non-selected fields is a good idea. If I have specified two mutually-exclusive options in a one-of situation, I feel like it would be safer to just error out, if we're not sure. At the very least, if the discriminator is not set and more than one option is populated, it should error. Your example `type: hostPath` should be HostPath. Will all such discriminators be optional forever, even for new fields, or will we require them in new one-of situations?

…

On Wed, Dec 21, 2016 at 6:06 PM, Mengqi Yu ***@***.***> wrote: Overview We have a number of cases in the API where only one of a set of fields is allowed to be specified, aka undiscriminated union / oneof. VolumeSource is the canonical example: it has fields such as emptyDir, gcePersistentDisk, awsElasticBlockStore and other 20 fields. Only one of these fields can be specified. We should add discriminators to union / oneof APIs, since it has several advantages. Original issue is described in kubernetes/kubernetes#35345 <kubernetes/kubernetes#35345> Advantages Adding discriminators to all unions/oneof cases would have multiple advantages: 1. Clients could effectively implement a switch instead of if-else trees to inspect the resource -- look at discriminator and lookup the corresponding field in a map (though differences in capitalization of the first letter in the API convention currently prevents the discriminator value from exactly matching the field name). 2. The API server could automatically clear non-selected fields, which would be convenient for kubectl apply and other cases. Analysis List of Impacted APIs In pkg/api/v1/types.go: - VolumeSource <https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/types.go#L235> - PersistentVolumeSource <https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/types.go#L345> - Handler <https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/types.go#L1485> - ContainerState <https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/types.go#L1576> - PodSignature <https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/types.go#L2973>: It has only one field, but the comment says "Exactly one field should be set". Maybe we will add more in the future? In pkg/authorization/types.go: - SubjectAccessReviewSpec <https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/authorization/types.go#L108>Exactly one of ResourceAttributes and NonResourceAttributes must be set. But other non-union fields also exist in this API. - SelfSubjectAccessReviewSpec <https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/authorization/types.go#L130> In pkg/apis/extensions/v1beta1/types.go: - DeploymentStrategy <https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/extensions/types.go#L249> - NetworkPolicyPeer <https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/extensions/v1beta1/types.go#L1188> - IngressRuleValue <https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/extensions/v1beta1/types.go#L733>: It says "exactly one of the following must be set". But it has only one field. Behavior If the discriminator were set, we'd require that the field corresponding to its value were set and the APIServer (registry) could automatically clear the other fields. If the discriminator were unset, behavior would be as before -- exactly one of the fields in the union/oneof would be required to be set and the operation would otherwise fail validation. We should set discriminators by default. This means we need to change it accordingly when the corresponding union/oneof fields were set and unset. If so, clients can rely on this for purpose (1). Proposed Changes API Add a discriminator field in all unions/oneof APIs. The discriminator should be optional for backward compatibility. There is an example below, the field Type works as a discriminator. type PersistentVolumeSource struct { // +optional GCEPersistentDisk *GCEPersistentDiskVolumeSource `json:"gcePersistentDisk,omitempty" protobuf:"bytes,1,opt,name=gcePersistentDisk"` // +optional AWSElasticBlockStore *AWSElasticBlockStoreVolumeSource `json:"awsElasticBlockStore,omitempty" protobuf:"bytes,2,opt,name=awsElasticBlockStore"` ... // Discriminator for PersistentVolumeSource, it can be "gcePersistentDisk", "awsElasticBlockStore" and etc. // +optional Type *string `json:"type,omitempty" protobuf:"bytes,24,opt,name=type"` } API Server We need to add defaulting logic described in the Behavior <#m_-5264161678971073059_behavior> section. kubectl No change required on kubectl. Example Discriminators Discriminator are Set by Default Assume we have added a field as discussed in section API changes <#m_-5264161678971073059_api> We first use kubectl apply -f create a PersistentVolume using the following config: apiVersion: v1kind: PersistentVolumemetadata: name: pv0001 annotations: volume.beta.kubernetes.io/storage-class: "slow"spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle hostPath: path: /tmp The subsequent kubectl get should have the discriminator field set by API server: apiVersion: v1kind: PersistentVolumemetadata: annotations: volume.beta.kubernetes.io/storage-class: slow creationTimestamp: 2016-12-22T00:56:31Z name: pv0001 namespace: "" resourceVersion: "1059564" selfLink: /api/v1/persistentvolumespv0001 uid: 7a57e42b-c7e1-11e6-aa89-42010a800002spec: accessModes: - ReadWriteOnce capacity: storage: 5Gi hostPath: path: /tmp persistentVolumeReclaimPolicy: Recycle # Discriminator showing the type is "hostPath" type: hostPathstatus: phase: Available Automatically Clear Unselected Fields Issue kubernetes/kubernetes#34292 <kubernetes/kubernetes#34292> will be fixed if spec.strategy.type is treated as a discriminator. Create a deployment using kubectl apply-f apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: nginx-deploymentspec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 Get the deployment back. Fields spec.strategy.type and spec.strategy.rollingUpdate have been defaulted. apiVersion: extensions/v1beta1kind: Deploymentmetadata: annotations: deployment.kubernetes.io/revision: "1" kubectl.kubernetes.io/last-applied-configuration: | {"kind":"Deployment","apiVersion":"extensions/v1beta1","metadata":{"name":"nginx-deployment","creationTimestamp":null},"spec":{"replicas":1,"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"nginx"}},"spec":{"containers":[{"name":"nginx","image":"nginx","ports":[{"containerPort":80}],"resources":{}}]}},"strategy":{}},"status":{}} creationTimestamp: 2016-12-22T01:23:34Z generation: 1 labels: app: nginx name: nginx-deployment namespace: default resourceVersion: "1062013" selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/nginx-deployment uid: 416ef165-c7e5-11e6-aa89-42010a800002spec: replicas: 1 selector: matchLabels: app: nginx strategy: // Defaulted by API server rollingUpdate: maxSurge: 1 maxUnavailable: 1 // Defaulted by API server type: RollingUpdate... Then we update the config file by adding strategy: type: Recreate Apply the new config by kubectl apply -f. The operation should succeed now, because now the API server knows to clear field spec.strategy.rollingUpdate after updating spec.strategy.type. @kubernetes/sig-api-machinery-misc <https://github.com/orgs/kubernetes/teams/sig-api-machinery-misc> @kubernetes/api-reviewers <https://github.com/orgs/kubernetes/teams/api-reviewers> @kubernetes/kubectl <https://github.com/orgs/kubernetes/teams/kubectl> — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#229>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVGCCzaFega2Ik6CS-1sNLezh4Xkuks5rKdsegaJpZM4LTkZW> .

[0xmichalis]

The API server could automatically clear non-selected fields

It's still not clear to me which is the appropriate place in the api server for doing this sort of defaulting. The rest handlers or where we set defaults?

cc: @kubernetes/sig-api-machinery-misc

[liggitt]

seems like repeating information in two places, and I agree that I'd rather get an error than have fields auto-cleared

[pwittrock]

Thanks for the comments.

The driving use case for this is the api server defaulting one-of-union values. The current behavior of providing an error message breaks kubectl apply's ability to update one-of fields defaulted by the server, as the values are missing from the original config and won't be cleared by apply.

We could make the union behavior configurable by a parameter to the api call so that apply can safely auto-clear server-defaulted union fields.

Thoughts on other ways can address the use case of clearing one-of fields defaulted by the server?

[thockin]

I was about to ask why 3-way doesn't catch this, but I think I get it. Blech. It's not set in the annotation... Are there any other cases where defaulting sets a value which the user might legitimately want to un-set it later? I can't think of any off the top of my head. I guess that argument makes this case much stronger. Thanks.

…

On Tue, Jan 3, 2017 at 10:50 AM, Phillip Wittrock ***@***.***> wrote: Thanks for the comments. The driving use case for this is the api server defaulting one-of-union values. The current behavior of providing an error message breaks kubectl apply's ability to update one-of fields defaulted by the server, as the values are missing from the original config and won't be cleared by apply. We could make the union behavior configurable by a parameter to the api call so that apply can safely auto-clear server-defaulted union fields. Thoughts on other ways can address the use case of clearing one-of fields defaulted by the server? — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVK69l2jBHun0-MTpFmt8CovLbhKNks5rOph1gaJpZM4LTkZW> .

[pwittrock]

Are there any other cases where defaulting sets a value which the user
might legitimately want to un-set it later? I can't think of any off the
top of my head.

I am not aware of any, but @ymqytw did an audit of all fields that are defaulted by the server that we could look at.

[pwittrock]

I also think it would be helpful to have this requirement defined as part of object / field definition instead of implementing in the object validation. Not a huge win, but would provide more consistency.

[lavalamp]

  hostPath:
    path: /tmp
  persistentVolumeReclaimPolicy: Recycle
  # Discriminator showing the type is "hostPath"
  type: hostPath

We can't just call the discriminators "type" if there's other fields mixed in. The above snippet is super confusing. These need to be nested one level deeper, or the name has to be relevant (e.g. mountType).

[thockin]

I'm not sure I grok the issue here? Barring some unfortunate situation where `type` was used as a name of a one-of option, why can't we standardize that name as the discriminator? ``` type Volume struct { Name string VolumeSource } type VolumeSource struct { Type string HostPath *HostPathVolumeSource EmptyDir *EmptyDirVolumeSource GCEPersistentDisk *GCEPersistentDiskVolumeSource //... } ```

…

On Fri, Jan 6, 2017 at 2:53 PM, Daniel Smith ***@***.***> wrote: hostPath: path: /tmp persistentVolumeReclaimPolicy: Recycle # Discriminator showing the type is "hostPath" type: hostPath We can't just call the discriminators "type" if there's other fields mixed in. The above snippet is super confusing. These need to be nested one level deeper, or the name has to be relevant (e.g. mountType). — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVADpW58sk65fiITC23ZnKIKP7PJDks5rPsXdgaJpZM4LTkZW> .

[lavalamp]

If it's nested without any siblings other than "type" and the one-of options, then you are right. But the example I quoted is not nested like that. (maybe it's a bug in the example, I haven't looked at the real object...

…

On Fri, Jan 6, 2017 at 5:26 PM, Tim Hockin ***@***.***> wrote: I'm not sure I grok the issue here? Barring some unfortunate situation where `type` was used as a name of a one-of option, why can't we standardize that name as the discriminator? ``` type Volume struct { Name string VolumeSource } type VolumeSource struct { Type string HostPath *HostPathVolumeSource EmptyDir *EmptyDirVolumeSource GCEPersistentDisk *GCEPersistentDiskVolumeSource //... } ``` On Fri, Jan 6, 2017 at 2:53 PM, Daniel Smith ***@***.***> wrote: > hostPath: > path: /tmp > persistentVolumeReclaimPolicy: Recycle > # Discriminator showing the type is "hostPath" > type: hostPath > > We can't just call the discriminators "type" if there's other fields mixed > in. The above snippet is super confusing. These need to be nested one level > deeper, or the name has to be relevant (e.g. mountType). > > — > You are receiving this because you are on a team that was mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/kubernetes/community/issues/ 229#issuecomment-271032327>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ AFVgVADpW58sk65fiITC23ZnKIKP7PJDks5rPsXdgaJpZM4LTkZW> > . > — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAnglrt5XFXbktun1-yTtVwEQzcC_g7Bks5rPunOgaJpZM4LTkZW> .

[thockin]

The example you quoted is exactly the one I highlighted, but I hear your concern that "type" is context dependent and may need to carry more meaning if the context is less obvious. On Fri, Jan 6, 2017 at 5:30 PM, Daniel Smith <notifications@github.com> wrote:

…

If it's nested without any siblings other than "type" and the one-of options, then you are right. But the example I quoted is not nested like that. (maybe it's a bug in the example, I haven't looked at the real object... On Fri, Jan 6, 2017 at 5:26 PM, Tim Hockin ***@***.***> wrote: > I'm not sure I grok the issue here? Barring some unfortunate situation > where `type` was used as a name of a one-of option, why can't we > standardize that name as the discriminator? > > ``` > type Volume struct { > Name string > VolumeSource > } > > type VolumeSource struct { > Type string > > HostPath *HostPathVolumeSource > EmptyDir *EmptyDirVolumeSource > GCEPersistentDisk *GCEPersistentDiskVolumeSource > //... > } > ``` > > On Fri, Jan 6, 2017 at 2:53 PM, Daniel Smith ***@***.***> > wrote: > > > hostPath: > > path: /tmp > > persistentVolumeReclaimPolicy: Recycle > > # Discriminator showing the type is "hostPath" > > type: hostPath > > > > We can't just call the discriminators "type" if there's other fields > mixed > > in. The above snippet is super confusing. These need to be nested one > level > > deeper, or the name has to be relevant (e.g. mountType). > > > > — > > You are receiving this because you are on a team that was mentioned. > > Reply to this email directly, view it on GitHub > > <https://github.com/kubernetes/community/issues/ > 229#issuecomment-271032327>, > > or mute the thread > > <https://github.com/notifications/unsubscribe-auth/ > AFVgVADpW58sk65fiITC23ZnKIKP7PJDks5rPsXdgaJpZM4LTkZW> > > . > > > > > — > You are receiving this because you are on a team that was mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/kubernetes/community/issues/ 229#issuecomment-271052082>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAnglrt5XFXbktun1- yTtVwEQzcC_g7Bks5rPunOgaJpZM4LTkZW> > . > — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVB0bp4g_oW9ZGtJMTuq7XQoZatK8ks5rPurJgaJpZM4LTkZW> .

[smarterclayton]

I'm a little uncomfortable with magic unsetting of values - I think this should be context sensitive:

1. PATCH setting two of a one of is an error
2. PATCH should replace a union when you specify a single member
3. UPDATE should behave like PATCH
4. 3-way should happen on the server (so naive clients don't have to implement union clearing)
5. No need for magic clearing?

[thockin]

Do we have a way to denote a one-of set via something like openapi?

…

On Fri, Jan 6, 2017 at 5:41 PM, Clayton Coleman ***@***.***> wrote: I'm a little uncomfortable with magic unsetting of values - I think this should be context sensitive: 1. PATCH setting two of a one of is an error 2. PATCH *should* replace a union when you specify a single member 3. UPDATE should behave like PATCH 4. 3-way should happen on the server (so naive clients don't have to implement union clearing) 5. No need for magic clearing? — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVB1V8aUOF2MeOZ5l560fnUzAFlE3ks5rPu1BgaJpZM4LTkZW> .

[pwittrock]

@smarterclayton

That makes a lot of sense to me. Similarly, we could introduce a struct tag patchStrategy:"replace" for maps and structs (as opposed to patchStrategy:"merge" for lists). Unions could be implemented by defining a field containing the set of Union values, and a replace strategy. This doesn't provide "1.", but should provide 2, 3 & 5. Semantically, this would be less meaningful than a tag denoting a "Union" specifically, but it is a simpler approach than trying to validate that a struct is in-fact a Union.

@thockin openapi supports extensions, so yes would could denote this through openapi. @mbohlool is more familiar with the details around this.

[thockin]

so something like: type PodVolume struct { Name string PodVolumeSource `openapi:oneof` } ?

…

On Mon, Jan 9, 2017 at 4:36 PM, Phillip Wittrock ***@***.***> wrote: @smarterclayton <https://github.com/smarterclayton> That makes a lot of sense to me. Similarly, we could introduce a struct tag patchStrategy:"replace" for maps and structs (as opposed to patchStrategy:"merge" for lists). Unions could be implemented by defining a field containing the set of Union values, and a replace strategy. This doesn't provide "1.", but should provide 2, 3 & 5. Semantically, this would be less meaningful than a tag denoting a "Union" specifically, but it is a simpler approach than trying to validate that a struct is in-fact a Union. @thockin <https://github.com/thockin> openapi supports extensions, so yes would could denote this through openapi. @mbohlool <https://github.com/mbohlool> is more familiar with the details around this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVM-ygiPaN-12nguDVx9Mkvp6JN--ks5rQtKrgaJpZM4LTkZW> .