Unable to use cloudfront wait invalidation-completed

[cw-sudo]

cloudfront wait invalidation-completed failed.
after waiting about 10 minutes , following error has occurred.

aws cloudfront wait invalidation-completed --distribution-id XXXXXXXXXXXXX --id XXXXXXXXXXXXX
Waiter InvalidationCompleted failed: Max attempts exceeded

and when I wait for completed invalidation , the same error has occurred.

My version: aws-cli/1.6.9 Python/2.7.6 Darwin/14.0.0

[kyleknap]

@cw-sudo

One quick way to figure out the issue is to run the waiter's implementation command (i.e. the command it uses to pull the status of the Invalidation). Based on the waiter model:
https://github.com/boto/botocore/blob/develop/botocore/data/aws/cloudfront/2014-10-21.waiters.json#L18

It will use cloudfront get-invalidation. You can then use this command to check the status of you invalidation. According to the model, the waiter will stop waiting once the Status member from the output of cloudfront get-invalidation is equal to Completed.

The error you are getting is that the waiter has called the get-invalidation command too many times with out seeing the Completed state, and as a result it errors out with a max attemps exceeded error.

If you don't mind the wait, could you run the wait command again and then paste the output of get-invalidation command. This will help me narrow down the issue.

[cw-sudo]

Hi @kyleknap
Thank you for your reply!

If you don't mind the wait, could you run the wait command again and then paste the output of get-invalidation command. This will help me narrow down the issue.

This is the result. Please confirm.

For new Invalidation (InProgress)

$ aws cloudfront create-invalidation --cli-input-json file://source.json                [16:12:39]
{
    "Invalidation": {
        "Status": "InProgress",
        "InvalidationBatch": {
            "Paths": {
                "Items": [
                    "/test/text1.txt"
                ],
                "Quantity": 1
            },
            "CallerReference": "20141213012r01"
        },
        "Id": "I300Y4OKSWIW6X",
        "CreateTime": "2014-12-13T07:12:56.645Z"
    },
    "Location": "https://cloudfront.amazonaws.com/2014-10-21/distribution/ETMA394YJ8V95/invalidation/I300Y4OKSWIW6X"
}

$ aws cloudfront get-invalidation --distribution-id ETMA394YJ8V95 --id I300Y4OKSWIW6X                [16:12:56]
{
    "Invalidation": {
        "Status": "InProgress",
        "InvalidationBatch": {
            "Paths": {
                "Items": [
                    "/test/text1.txt"
                ],
                "Quantity": 1
            },
            "CallerReference": "20141213012r01"
        },
        "Id": "I300Y4OKSWIW6X",
        "CreateTime": "2014-12-13T07:12:56.645Z"
    }
}

$ aws cloudfront wait invalidation-completed --distribution-id ETMA394YJ8V95 --id I300Y4OKSWIW6X                [16:14:25]

Waiter InvalidationCompleted failed: Max attempts exceeded
FAIL: 255

$ aws cloudfront get-invalidation --distribution-id ETMA394YJ8V95 --id I300Y4OKSWIW6X                           [16:24:35]
{
    "Invalidation": {
        "Status": "InProgress",
        "InvalidationBatch": {
            "Paths": {
                "Items": [
                    "/test/text1.txt"
                ],
                "Quantity": 1
            },
            "CallerReference": "20141213012r01"
        },
        "Id": "I300Y4OKSWIW6X",
        "CreateTime": "2014-12-13T07:12:56.645Z"
    }
}

For Completed Invalidation (Completed)

$ aws cloudfront get-invalidation --distribution-id ETMA394YJ8V95 --id I300Y4OKSWIW6X                 [16:31:14]
{
    "Invalidation": {
        "Status": "Completed",
        "InvalidationBatch": {
            "Paths": {
                "Items": [
                    "/test/text1.txt"
                ],
                "Quantity": 1
            },
            "CallerReference": "20141213012r01"
        },
        "Id": "I300Y4OKSWIW6X",
        "CreateTime": "2014-12-13T07:12:56.645Z"
    }
}

$ aws cloudfront wait invalidation-completed --distribution-id ETMA394YJ8V95 --id I300Y4OKSWIW6X      [16:33:26]

Waiter InvalidationCompleted failed: Max attempts exceeded
FAIL: 255

$ aws cloudfront get-invalidation --distribution-id ETMA394YJ8V95 --id I300Y4OKSWIW6X                 [16:43:20]
{
    "Invalidation": {
        "Status": "Completed",
        "InvalidationBatch": {
            "Paths": {
                "Items": [
                    "/test/text1.txt"
                ],
                "Quantity": 1
            },
            "CallerReference": "20141213012r01"
        },
        "Id": "I300Y4OKSWIW6X",
        "CreateTime": "2014-12-13T07:12:56.645Z"
    }
}

[kyleknap]

@cw-sudo thanks for the output! Let me tell you what is going on. It looks like there are two issues here both related to our waiter model for cloudfront, https://github.com/boto/botocore/blob/develop/botocore/data/aws/cloudfront/2014-10-21.waiters.json#L18.

In deciding whether to stop waiting, our model tells the CLI what element to look at in the response to determine if we need to continue waiting. Currently, the model tells the CLI to look for the if "Status" element is equal to "Completed". However, that is incorrect because there is the top level "Invalidation" member in the JSON. We should actually be looking at "Invalidation.Status" member.

As to the second issue, it looks like even if the first item is fixed, it will still affect the case where the waiter times out and the invalidation is still in progress. Based on the output shown in the invalidation InProgess example, the waiter is hitting its timeout before it reaches the completed state. This should not happen. To fix it, we need to up either the number of retries or the length in which we delay.

We will get this fixed soon.

[kyleknap]

This should be fixed via boto/botocore#426. Closing issue. Please reopen, if the issue persists.