Kubernetes: Unable to Disable/Destroy cluster

[dgarbus]

Environment

I am running Spinnaker from the provided AMI and updated packages using apt. The versions I am currently using are as follows:

ii  spinnaker                        0.36.0-3                         all          spinnaker
ii  spinnaker-clouddriver            1.143.0-4                        all          spinnaker-clouddriver
ii  spinnaker-deck                   2.587.0-3                        all          spinnaker-deck
ii  spinnaker-echo                   1.99.0-3                         all          spinnaker-echo
ii  spinnaker-front50                1.17.0-3                         all          spinnaker-front50
ii  spinnaker-gate                   2.45.0-3                         all          spinnaker-gate
ii  spinnaker-igor                   1.22.0-3                         all          spinnaker-igor
ii  spinnaker-orca                   1.103.0-3                        all          spinnaker-orca
ii  spinnaker-rosco                  0.30.0-3                         all          spinnaker-rosco
ii  spinnaker-rush                   1.25.0-3                         all          spinnaker-rush

For Kubernetes, I am running a CoreOS k8s deployment using kube-aws. I have dropped the config file and all certificates under /home/spinnaker/.kube on the Spinnaker host. I have also confirmed that Spinnaker is able to connect to Kubernetes and successfully create pods.

Steps to Reproduce

1. Create a new application in Spinnaker.
2. Add a new server group within the application (Kubernetes provider).
3. Select a container image and the necessary requirements to launch the container.
4. Save config and confirm that cluster comes up successfully.
5. Select your new cluster and attempt to disable or destroy it from the Actions menu.

Additional Details

Disabling/Destroying will fail on the 'Monitor Server Group' step with the following messages in the clouddriver log. The pods will not be cleaned up on the Kubernetes side.

2016-04-27 18:31:04.181  INFO 4598 --- [pool-3-thread-1] c.n.s.c.data.task.jedis.JedisTask        : [ORCHESTRATION] - Processing op: DisableKubernetesAtomicOperation
2016-04-27 18:31:04.192  INFO 4598 --- [pool-3-thread-1] c.n.s.c.data.task.jedis.JedisTask        : [DISABLE] - Initializing disable operation...
2016-04-27 18:31:04.193  INFO 4598 --- [pool-3-thread-1] c.n.s.c.data.task.jedis.JedisTask        : [DISABLE] - Looking up provided namespace...
2016-04-27 18:31:04.196  INFO 4598 --- [pool-3-thread-1] c.n.s.c.data.task.jedis.JedisTask        : [DISABLE] - Finding requisite replication controller...
2016-04-27 18:31:04.202  INFO 4598 --- [pool-3-thread-1] c.n.s.c.data.task.jedis.JedisTask        : [DISABLE] - Getting list of attached services...
2016-04-27 18:31:04.205  INFO 4598 --- [pool-3-thread-1] c.n.s.c.data.task.jedis.JedisTask        : [DISABLE] - Resetting replication controller service template labels and selectors...
2016-04-27 18:31:05.363  INFO 4598 --- [pool-3-thread-1] c.n.s.c.data.task.jedis.JedisTask        : [DISABLE] - Finding affected pods...
2016-04-27 18:31:05.377  INFO 4598 --- [pool-3-thread-1] c.n.s.c.data.task.jedis.JedisTask        : [DISABLE] - Resetting service labels for each pod...
2016-04-27 18:31:05.717 ERROR 4598 --- [pool-3-thread-1] c.n.s.c.k.api.KubernetesApiAdaptor       : [io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestException(OperationSupport.java:299), io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:222), io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:200), io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:506), io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:82), io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation$1.apply(HasMetadataOperation.java:50), io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation$1.apply(HasMetadataOperation.java:40), io.fabric8.kubernetes.api.model.DoneablePod.done(DoneablePod.java:31), io.fabric8.kubernetes.api.model.DoneablePod.done(DoneablePod.java:11), io.fabric8.kubernetes.api.model.Doneable$done.call(Unknown Source), org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45), org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108), org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112), com.netflix.spinnaker.clouddriver.kubernetes.api.KubernetesApiAdaptor.togglePodLabels(KubernetesApiAdaptor.groovy:197), com.netflix.spinnaker.clouddriver.kubernetes.api.KubernetesApiAdaptor$togglePodLabels$9.call(Unknown Source), org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45), org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108), org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128), com.netflix.spinnaker.clouddriver.kubernetes.deploy.ops.servergroup.AbstractEnableDisableKubernetesAtomicOperation$_operate_closure2.doCall(AbstractEnableDisableKubernetesAtomicOperation.groovy:79), sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method), sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62), sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.lang.reflect.Method.invoke(Method.java:498), org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90), groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324), org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:292), groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1015), groovy.lang.Closure.call(Closure.java:423), org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:51), org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:103), com.sun.proxy.$Proxy109.accept(Unknown Source), java.util.ArrayList.forEach(ArrayList.java:1249), sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source), sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.lang.reflect.Method.invoke(Method.java:498), org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrap.invoke(PojoMetaMethodSite.java:210), org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53), org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45), org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108), org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116), com.netflix.spinnaker.clouddriver.kubernetes.deploy.ops.servergroup.AbstractEnableDisableKubernetesAtomicOperation.operate(AbstractEnableDisableKubernetesAtomicOperation.groovy:74), com.netflix.spinnaker.clouddriver.kubernetes.deploy.ops.servergroup.DisableKubernetesAtomicOperation.super$2$operate(DisableKubernetesAtomicOperation.groovy), sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method), sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62), sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.lang.reflect.Method.invoke(Method.java:498), org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90), groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324), groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1206), org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:130), com.netflix.spinnaker.clouddriver.kubernetes.deploy.ops.servergroup.DisableKubernetesAtomicOperation.operate(DisableKubernetesAtomicOperation.groovy:46), com.netflix.spinnaker.clouddriver.kubernetes.deploy.ops.servergroup.DisableKubernetesAtomicOperation.operate(DisableKubernetesAtomicOperation.groovy), com.netflix.spinnaker.clouddriver.orchestration.AtomicOperation$operate.call(Unknown Source), org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45), org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108), org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116), com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1_closure2.doCall(DefaultOrchestrationProcessor.groovy:65), com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1_closure2.doCall(DefaultOrchestrationProcessor.groovy), sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method), sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62), sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.lang.reflect.Method.invoke(Method.java:498), org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90), groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324), org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:292), groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1015), groovy.lang.Closure.call(Closure.java:423), groovy.lang.Closure.call(Closure.java:417), com.netflix.spinnaker.clouddriver.metrics.TimedCallable$ClosureWrapper.call(TimedCallable.groovy:55), com.netflix.spinnaker.clouddriver.metrics.TimedCallable.call(TimedCallable.groovy:82), java_util_concurrent_Callable$call.call(Unknown Source), org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45), org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108), org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112), com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1.doCall(DefaultOrchestrationProcessor.groovy:64), com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1.doCall(DefaultOrchestrationProcessor.groovy), sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method), sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62), sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.lang.reflect.Method.invoke(Method.java:498), org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90), groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324), org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:292), groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1015), groovy.lang.Closure.call(Closure.java:423), groovy.lang.Closure.call(Closure.java:417), com.netflix.spinnaker.clouddriver.metrics.TimedCallable$ClosureWrapper.call(TimedCallable.groovy:55), com.netflix.spinnaker.clouddriver.metrics.TimedCallable.call(TimedCallable.groovy:82), java.util.concurrent.FutureTask.run(FutureTask.java:266), java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142), java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617), java.lang.Thread.run(Thread.java:745)]
2016-04-27 18:31:05.733  INFO 4598 --- [pool-3-thread-1] c.n.s.c.data.task.jedis.JedisTask        : [ORCHESTRATION] - Orchestration failed: DisableKubernetesAtomicOperation | KubernetesOperationException: [Toggle Pod Labels failed: Error executing: PUT at: https://kubernetes-test.aws.example.com/api/v1/namespaces/default/pods/helloworld-v000-w1oxk. Cause: org.codehaus.groovy.runtime.GStringImpl cannot be cast to java.lang.String]
2016-04-27 18:31:05.739 ERROR 4598 --- [pool-3-thread-1] c.n.s.c.o.DefaultOrchestrationProcessor  : com.netflix.spinnaker.clouddriver.kubernetes.deploy.exception.KubernetesOperationException: Toggle Pod Labels failed: Error executing: PUT at: https://kubernetes-test.aws.example.com/api/v1/namespaces/default/pods/helloworld-v000-w1oxk. Cause: org.codehaus.groovy.runtime.GStringImpl cannot be cast to java.lang.String
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
    at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:194)
    at com.netflix.spinnaker.clouddriver.kubernetes.api.KubernetesApiAdaptor.togglePodLabels(KubernetesApiAdaptor.groovy:200)
    at com.netflix.spinnaker.clouddriver.kubernetes.api.KubernetesApiAdaptor$togglePodLabels$9.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
    at com.netflix.spinnaker.clouddriver.kubernetes.deploy.ops.servergroup.AbstractEnableDisableKubernetesAtomicOperation$_operate_closure2.doCall(AbstractEnableDisableKubernetesAtomicOperation.groovy:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:292)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1015)
    at groovy.lang.Closure.call(Closure.java:423)
    at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:51)
    at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:103)
    at com.sun.proxy.$Proxy109.accept(Unknown Source)
    at java.util.ArrayList.forEach(ArrayList.java:1249)
    at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrap.invoke(PojoMetaMethodSite.java:210)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    at com.netflix.spinnaker.clouddriver.kubernetes.deploy.ops.servergroup.AbstractEnableDisableKubernetesAtomicOperation.operate(AbstractEnableDisableKubernetesAtomicOperation.groovy:74)
    at com.netflix.spinnaker.clouddriver.kubernetes.deploy.ops.servergroup.DisableKubernetesAtomicOperation.super$2$operate(DisableKubernetesAtomicOperation.groovy)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1206)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:130)
    at com.netflix.spinnaker.clouddriver.kubernetes.deploy.ops.servergroup.DisableKubernetesAtomicOperation.operate(DisableKubernetesAtomicOperation.groovy:46)
    at com.netflix.spinnaker.clouddriver.kubernetes.deploy.ops.servergroup.DisableKubernetesAtomicOperation.operate(DisableKubernetesAtomicOperation.groovy)
    at com.netflix.spinnaker.clouddriver.orchestration.AtomicOperation$operate.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1_closure2.doCall(DefaultOrchestrationProcessor.groovy:65)
    at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1_closure2.doCall(DefaultOrchestrationProcessor.groovy)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:292)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1015)
    at groovy.lang.Closure.call(Closure.java:423)
    at groovy.lang.Closure.call(Closure.java:417)
    at com.netflix.spinnaker.clouddriver.metrics.TimedCallable$ClosureWrapper.call(TimedCallable.groovy:55)
    at com.netflix.spinnaker.clouddriver.metrics.TimedCallable.call(TimedCallable.groovy:82)
    at java_util_concurrent_Callable$call.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
    at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1.doCall(DefaultOrchestrationProcessor.groovy:64)
    at com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1.doCall(DefaultOrchestrationProcessor.groovy)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:292)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1015)
    at groovy.lang.Closure.call(Closure.java:423)
    at groovy.lang.Closure.call(Closure.java:417)
    at com.netflix.spinnaker.clouddriver.metrics.TimedCallable$ClosureWrapper.call(TimedCallable.groovy:55)
    at com.netflix.spinnaker.clouddriver.metrics.TimedCallable.call(TimedCallable.groovy:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

[lwander]

I started looking into this today, and found the cause of the funky message: spinnaker/clouddriver#580 (the error handler catching a 403 was converting a string incorrectly). We still can't toggle pod labels for k8s installations on AWS, but I'm reaching out the client library about that next. See the PR for context.

[lwander]

Filed a bug: fabric8io/kubernetes-client#396

[lwander]

Looks like the latest fabric8 api fixes this issue, going to push a clouddriver release in a minute with the changes. Would appreciate if someone else can verify to double check I've replicated this properly.

[lwander]

Release is here, should be available in ~20 minutes https://github.com/spinnaker/clouddriver/releases/tag/v1.216.0

[lwander]

(Not sure why github auto-closed this issue)

[cocotton]

It works for me, now able to delete clusters. Thanks!

[makii42]

👍 Ran in the same issue last night, and already fixed! Thanks!