cloudformation module no longer works in 2.3.0 #23593

jagibson · 2017-04-13T20:18:56Z

ISSUE TYPE

Bug Report

COMPONENT NAME

cloudformation.py

ANSIBLE VERSION

ansible --version
ansible 2.3.0.0
  config file = /home/jagibson/ansible/playbooks/myplaybook/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.12+ (default, Sep 17 2016, 12:08:02) [GCC 6.2.0 20160914]

CONFIGURATION

ansible.cfg

[defaults]
roles_path = roles
remote_user = ubuntu
host_key_checking = false
retry_files_enabled = False
inventory = inventory
jinja2_extensions = jinja2.ext.do

[ssh_connection]
pipelining = True
retries = 5

OS / ENVIRONMENT

Ubuntu 16.10
ansible (2.3.0.0)
boto (2.46.1)
boto3 (1.4.4)
botocore (1.5.39)

SUMMARY

When running the cloudformation module in Ansible 2.3.0 the play fails with the error:

"botocore.exceptions.ClientError: An error occurred (InvalidAction) when calling the DescribeStacks operation: The action DescribeStacks is not valid for this web service."

When running Ansible 2.2.1 the module works as expected.

STEPS TO REPRODUCE

test.yml

---
- hosts: localhost
  gather_facts: false
  connection: local
  vars:
    eni_stack_name: dev-host1-us-east-1-a
    eni_tags: 
      Name: "dev_eni"
  tasks:
    - name: run cloudformation
      cloudformation:
        stack_name: "{{ eni_stack_name }}"
        template: "tmp/{{ eni_stack_name }}.yml"
        tags: "{{ eni_tags }}"
      register: eni_cf_outputs

tmp/dev-host1-us-east-1-a.yml

---
AWSTemplateFormatVersion: 2010-09-09
Description: Creates an Amazon ENI, DNS Records, and associated security groups.
Resources:
  ENI:
    Type: AWS::EC2::NetworkInterface
    Properties:
      Description: Elastic Network Interface
      GroupSet:
        - sg-01234567
        - Ref: ENISG
      SubnetId: "subnet-01234567"
      SourceDestCheck: "true"
      PrivateIpAddress: "10.20.30.40"
      Tags:
        - Key: "Name"
          Value: "dev_eni"
  ENISG:
    Type: "AWS::EC2::SecurityGroup"
    Properties: 
      GroupDescription: Primary security group
      SecurityGroupIngress:
        - CidrIp: "10.20.30.0/24"
          FromPort: "22"
          ToPort: "22"
          IpProtocol: "tcp"
      Tags:
        - Key: "Name"
          Value: "dev_eni"
      VpcId: "vpc-01234567"
  ForwardDNS:
    Type: "AWS::Route53::RecordSet"
    Properties: 
      HostedZoneName: "myinternal.zone."
      Name: "dev-host1-us-east-1-a.myinternal.zone."
      ResourceRecords:
        - Fn::GetAtt:
            - ENI
            - PrimaryPrivateIpAddress
      TTL: "300"
      Type: "A"
Outputs:
  EniIp:
    Description: Primary IP address of the ENI
    Value: 
      Fn::GetAtt:
        - ENI
        - PrimaryPrivateIpAddress
  EniId:
    Description: AWS ID for the ENI
    Value:
      Ref: ENI
  EniSg:
    Description: Primary security group ID for the ENI
    Value:
      Fn::GetAtt:
        - ENISG
        - GroupId
  EniDns:
    Description: DNS record for the ENI
    Value:
      Ref: ForwardDNS

EXPECTED RESULTS

Ansible runs successfully
(here's the 2.2.1 output)

ansible-playbook test.yml -vvvv
Using /home/jagibson/ansible/playbooks/myplaybook/ansible.cfg as config file
Loading callback plugin default of type stdout, v2.0 from/home/jagibson/ansible/playbooks/myplaybook/.virtualenv/local/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: test.yml *************************************************************
1 plays in test.yml

PLAY [localhost] ***************************************************************

TASK [run cloudformation] ******************************************************
task path: /home/jagibson/ansible/playbooks/myplaybook/test.yml:10
Using module file /home/jagibson/ansible/playbooks/myplaybook/.virtualenv/local/lib/python2.7/site-packages/ansible/modules/core/cloud/amazon/cloudformation.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: jgibson
<127.0.0.1> EXEC /bin/sh -c '/home/jagibson/ansible/playbooks/myplaybook/.virtualenv/bin/python2 && sleep 0'
changed: [localhost] => {
    "changed": true, 
    "events": [
        "StackEvent AWS::CloudFormation::Stack dev-host1-us-east-1-a CREATE_COMPLETE", 
        "StackEvent AWS::Route53::RecordSet ForwardDNS CREATE_COMPLETE", 
        "StackEvent AWS::Route53::RecordSet ForwardDNS CREATE_IN_PROGRESS", 
        "StackEvent AWS::Route53::RecordSet ForwardDNS CREATE_IN_PROGRESS", 
        "StackEvent AWS::EC2::NetworkInterface ENI CREATE_COMPLETE", 
        "StackEvent AWS::EC2::NetworkInterface ENI CREATE_IN_PROGRESS", 
        "StackEvent AWS::EC2::NetworkInterface ENI CREATE_IN_PROGRESS", 
        "StackEvent AWS::EC2::SecurityGroup ENISG CREATE_COMPLETE", 
        "StackEvent AWS::EC2::SecurityGroup ENISG CREATE_IN_PROGRESS", 
        "StackEvent AWS::EC2::SecurityGroup ENISG CREATE_IN_PROGRESS", 
        "StackEvent AWS::CloudFormation::Stack dev-host1-us-east-1-a CREATE_IN_PROGRESS"
    ], 
    "invocation": {
        "module_args": {
            "aws_access_key": null, 
            "aws_secret_key": null, 
            "disable_rollback": false, 
            "ec2_url": null, 
            "notification_arns": null, 
            "profile": null, 
            "region": null, 
            "security_token": null, 
            "stack_name": "dev-host1-us-east-1-a", 
            "stack_policy": null, 
            "state": "present", 
            "tags": {
                "Name": "dev_eni"
            }, 
            "template": "tmp/dev-host1-us-east-1-a.yml", 
            "template_format": null, 
            "template_parameters": {}, 
            "template_url": null, 
            "validate_certs": true
        }, 
        "module_name": "cloudformation"
    }, 
    "output": "Stack CREATE complete", 
    "stack_outputs": {
        "EniDns": "dev-host1-us-east-1-a.myinternal.zone", 
        "EniId": "eni-01234567", 
        "EniIp": "10.20.30.40", 
        "EniSg": "sg-01234567"
    }, 
    "stack_resources": [
        {
            "last_updated_time": null, 
            "logical_resource_id": "ENI", 
            "physical_resource_id": "eni-01234567", 
            "resource_type": "AWS::EC2::NetworkInterface", 
            "status": "CREATE_COMPLETE", 
            "status_reason": null
        }, 
        {
            "last_updated_time": null, 
            "logical_resource_id": "ENISG", 
            "physical_resource_id": "sg-01234567", 
            "resource_type": "AWS::EC2::SecurityGroup", 
            "status": "CREATE_COMPLETE", 
            "status_reason": null
        }, 
        {
            "last_updated_time": null, 
            "logical_resource_id": "ForwardDNS", 
            "physical_resource_id": "dev-host1-us-east-1-a.myinternal.zone", 
            "resource_type": "AWS::Route53::RecordSet", 
            "status": "CREATE_COMPLETE", 
            "status_reason": null
        }
    ]
}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0

ACTUAL RESULTS

ansible-playbook test.yml -vvvv
Using /home/jagibson/ansible/playbooks/myplaybook/ansible.cfg as config file

Loading callback plugin default of type stdout, v2.0 from /home/jagibson/ansible/playbooks/myplaybook/.virtualenv/local/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: test.yml *******************************************************************************************************************************************************
1 plays in test.yml

PLAY [localhost] *********************************************************************************************************************************************************
META: ran handlers

TASK [run cloudformation] ************************************************************************************************************************************************
task path: /home/jagibson/ansible/playbooks/myplaybook/test.yml:10
Using module file /home/jagibson/ansible/playbooks/myplaybook/.virtualenv/local/lib/python2.7/site-packages/ansible/modules/cloud/amazon/cloudformation.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: jgibson
<127.0.0.1> EXEC /bin/sh -c '/home/jagibson/ansible/playbooks/myplaybook/.virtualenv/bin/python2 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_dpM0mB/ansible_module_cloudformation.py", line 508, in <module>
    main()
  File "/tmp/ansible_dpM0mB/ansible_module_cloudformation.py", line 452, in main
    stack_info = get_stack_facts(cfn, stack_params['StackName'])
  File "/tmp/ansible_dpM0mB/ansible_module_cloudformation.py", line 381, in get_stack_facts
    raise err
botocore.exceptions.ClientError: An error occurred (InvalidAction) when calling the DescribeStacks operation: The action DescribeStacks is not valid for this web service.

fatal: [localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_dpM0mB/ansible_module_cloudformation.py\", line 508, in <module>\n    main()\n  File \"/tmp/ansible_dpM0mB/ansible_module_cloudformation.py\", line 452, in main\n    stack_info = get_stack_facts(cfn, stack_params['StackName'])\n  File \"/tmp/ansible_dpM0mB/ansible_module_cloudformation.py\", line 381, in get_stack_facts\n    raise err\nbotocore.exceptions.ClientError: An error occurred (InvalidAction) when calling the DescribeStacks operation: The action DescribeStacks is not valid for this web service.\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}

PLAY RECAP ***************************************************************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1

The text was updated successfully, but these errors were encountered:

ansibot · 2017-04-13T20:26:44Z

cc @tedder @ryansb
click here for bot help

jagibson · 2017-04-13T22:43:25Z

Update - I'm seeing a similar error trying to use the cloudformation_facts module, even with Ansible 2.2.1. Perhaps it is related to boto3?

howinator · 2017-04-14T15:44:19Z

Are you getting this error for creating or updating a stack? Or both?

jagibson · 2017-04-14T17:46:03Z

@howinator Both. It may be related to Ubuntu versions sometime after 14.04. I tested on Ubuntu 14.04.4 and at least the cloudformation_facts module works.

tedder · 2017-04-15T00:16:10Z

Does this work?

aws cloudformation describe-stacks --max-items 1

Note 2.3 has a rewritten cloudformation module, fully in boto3. I use Ubuntu 14.04 and 16.04, no problems.

s-hertel · 2017-04-18T15:58:10Z

I created a Ubuntu instance and was able to run your reproducer without any errors. Maybe it's specific to your setup?

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.10
Release:	16.10
Codename:	yakket

ansible --version
ansible 2.3.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.12+ (default, Sep 17 2016, 12:08:02) [GCC 6.2.0 20160914]

pip freeze | grep boto
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
boto3==1.4.4
botocore==1.5.39

ansible-playbook test.yml -vvvvvvvv
Using /etc/ansible/ansible.cfg as config file
 [WARNING]: provided hosts list is empty, only localhost is available

Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: test.yml ********************************************************************************************************************
1 plays in test.yml

PLAY [localhost] **********************************************************************************************************************
META: ran handlers

TASK [run cloudformation] *************************************************************************************************************
task path: /home/ubuntu/test.yml:10
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/cloud/amazon/cloudformation.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: ubuntu
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/ubuntu/.ansible/tmp/ansible-tmp-1492530525.58-129871967093928 `" && echo ansible-tmp-1492530525.58-129871967093928="` echo /home/ubuntu/.ansible/tmp/ansible-tmp-1492530525.58-129871967093928 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmpPlamGK TO /home/ubuntu/.ansible/tmp/ansible-tmp-1492530525.58-129871967093928/cloudformation.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/ubuntu/.ansible/tmp/ansible-tmp-1492530525.58-129871967093928/ /home/ubuntu/.ansible/tmp/ansible-tmp-1492530525.58-129871967093928/cloudformation.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/ubuntu/.ansible/tmp/ansible-tmp-1492530525.58-129871967093928/cloudformation.py; rm -rf "/home/ubuntu/.ansible/tmp/ansible-tmp-1492530525.58-129871967093928/" > /dev/null 2>&1 && sleep 0'
changed: [localhost] => {
    "changed": true, 
    "events": [
        "StackEvent AWS::CloudFormation::Stack dev-host1-us-east-1-a CREATE_COMPLETE", 
        "StackEvent AWS::Route53::RecordSet ForwardDNS CREATE_COMPLETE", 
        "StackEvent AWS::Route53::RecordSet ForwardDNS CREATE_IN_PROGRESS", 
        "StackEvent AWS::Route53::RecordSet ForwardDNS CREATE_IN_PROGRESS", 
        "StackEvent AWS::EC2::NetworkInterface ENI CREATE_COMPLETE", 
        "StackEvent AWS::EC2::NetworkInterface ENI CREATE_IN_PROGRESS", 
        "StackEvent AWS::EC2::NetworkInterface ENI CREATE_IN_PROGRESS", 
        "StackEvent AWS::EC2::SecurityGroup ENISG CREATE_COMPLETE", 
        "StackEvent AWS::EC2::SecurityGroup ENISG CREATE_IN_PROGRESS", 
        "StackEvent AWS::EC2::SecurityGroup ENISG CREATE_IN_PROGRESS", 
        "StackEvent AWS::CloudFormation::Stack dev-host1-us-east-1-a CREATE_IN_PROGRESS"
    ], 
    "invocation": {
        "module_args": {
            "aws_access_key": null, 
            "aws_secret_key": null, 
            "disable_rollback": false, 
            "ec2_url": null, 
            "notification_arns": null, 
            "profile": null, 
            "region": null, 
            "role_arn": null, 
            "security_token": null, 
            "stack_name": "dev-host1-us-east-1-a", 
            "stack_policy": null, 
            "state": "present", 
            "tags": {
                "Name": "dev_eni"
            }, 
            "template": "tmp/dev-host1-us-east-1-a.yml", 
            "template_format": null, 
            "template_parameters": {}, 
            "template_url": null, 
            "validate_certs": true
        }
    }, 
    "log": [], 
    "output": "Stack CREATE complete", 
    "stack_outputs": {
        "EniDns": "dev-host1-us-east-1-a.ip-172-31-15-194.ec2.internal", 
        "EniId": "eni-46d14ba3", 
        "EniIp": "172.31.15.37", 
        "EniSg": "sg-7dc5e702"
    }, 
    "stack_resources": [
        {
            "last_updated_time": "2017-04-18T15:49:37.680000+00:00", 
            "logical_resource_id": "ENI", 
            "physical_resource_id": "eni-46d14ba3", 
            "resource_type": "AWS::EC2::NetworkInterface", 
            "status": "CREATE_COMPLETE", 
            "status_reason": null
        }, 
        {
            "last_updated_time": "2017-04-18T15:49:15.558000+00:00", 
            "logical_resource_id": "ENISG", 
            "physical_resource_id": "sg-7dc5e702", 
            "resource_type": "AWS::EC2::SecurityGroup", 
            "status": "CREATE_COMPLETE", 
            "status_reason": null
        }, 
        {
            "last_updated_time": "2017-04-18T15:50:44.081000+00:00", 
            "logical_resource_id": "ForwardDNS", 
            "physical_resource_id": "dev-host1-us-east-1-a.ip-172-31-15-194.ec2.internal", 
            "resource_type": "AWS::Route53::RecordSet", 
            "status": "CREATE_COMPLETE", 
            "status_reason": null
        }
    ]
}
META: ran handlers
META: ran handlers

PLAY RECAP ****************************************************************************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0

tedder · 2017-04-18T17:13:35Z

needs_info. Can't proceed without the requested information.

jagibson · 2017-04-21T18:45:33Z

@tedder @s-hertel Yeah I'm starting to suspect something is wrong with my setup. Researching.

mixja · 2017-05-01T21:23:40Z

I am getting the following exception from the module - as per Boto3 docs all parameters must be passed as a string, looking at the current CloudFormation module code, there is no cast to a string.

In Ansible 2.2 and prior, same approach was taken but call was made to a Boto provider which I suspect did this automatically for you.

TASK [aws-cloudformation : configure application stack] ***************************************************************************************************************
fatal: [dev]: FAILED! => {"changed": false, "failed": true, "msg": "Parameter validation failed:\nInvalid type for parameter Parameters[4].ParameterValue, value: 10, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[14].ParameterValue, value: 9200, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[17].ParameterValue, value: 3000, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[19].ParameterValue, value: 443, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[22].ParameterValue, value: 2, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[24].ParameterValue, value: 30, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[26].ParameterValue, value: 1, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[31].ParameterValue, value: 9200, type: <type 'int'>, valid types: <type 'basestring'> Parameter validation failed:\nInvalid type for parameter Parameters[4].ParameterValue, value: 10, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[14].ParameterValue, value: 9200, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[17].ParameterValue, value: 3000, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[19].ParameterValue, value: 443, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[22].ParameterValue, value: 2, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[24].ParameterValue, value: 30, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[26].ParameterValue, value: 1, type: <type 'int'>, valid types: <type 'basestring'>\nInvalid type for parameter Parameters[31].ParameterValue, value: 9200, type: <type 'int'>, valid types: <type 'basestring'> - <class 'botocore.exceptions.ParamValidationError'>"}

mixja · 2017-05-01T21:38:51Z

Confirming changing this line of code to the following:

stack_params['Parameters'] = [{'ParameterKey':k, 'ParameterValue':str(v)} for k, v in template_parameters.items()]

seems to fix the problem

s-hertel · 2017-05-02T13:30:58Z

@jagibson Hi, can you answer tedder's question using the AWS CLI and if that works, can you test mixja's fix, please?

jagibson · 2017-05-04T21:51:06Z

@s-hertel Yes, but I probably won't be able to get to it until next week.

Update: the 'aws cloudformation describe-stacks --max-items 1' command works fine

tedder · 2017-05-04T22:23:50Z

notabug

…nsible#24182)

#23593 (#25051)

jagibson · 2017-07-13T22:50:38Z

Just as a follow-up - I've since moved to a new laptop and am running Ansible 2.3.1. I've installed all of my python modules inside of a virtualenv environment (besides the os defaults) and I'm not getting the error anymore.

ansibot added affects_2.3 This issue/PR affects Ansible v2.3 aws bug_report cloud module This issue/PR relates to a module. needs_triage Needs a first human triage before being processed. labels Apr 13, 2017

ansibot assigned ryansb Apr 13, 2017

nitzmahone removed the needs_triage Needs a first human triage before being processed. label Apr 13, 2017

ansibot added the needs_info This issue requires further information. Please answer any outstanding questions. label Apr 18, 2017

ansibot removed the needs_info This issue requires further information. Please answer any outstanding questions. label Apr 21, 2017

s-hertel added the needs_info This issue requires further information. Please answer any outstanding questions. label Apr 22, 2017

mixja added a commit to mixja/ansible that referenced this issue May 1, 2017

Cast CloudFormation parameter values to string - fixes ansible#23593

02410fc

mixja mentioned this issue May 1, 2017

Cast CloudFormation parameter values to string #24182

Merged

ansibot removed the needs_info This issue requires further information. Please answer any outstanding questions. label May 4, 2017

ansibot closed this as completed May 4, 2017

s-hertel pushed a commit that referenced this issue May 15, 2017

Cast CloudFormation parameter values to string - fixes #23593 (#24182)

8dae2ea

KKoukiou pushed a commit to KKoukiou/ansible that referenced this issue May 22, 2017

Cast CloudFormation parameter values to string - fixes ansible#23593 (a…

7d6b472

…nsible#24182)

therealdwright pushed a commit to therealdwright/ansible that referenced this issue May 25, 2017

Cast CloudFormation parameter values to string - fixes ansible#23593

ed1645a

ryansb pushed a commit that referenced this issue May 26, 2017

[cloud] Backport handling of CloudFormation int parameter values - fixes

f829d58

#23593 (#25051)

jimi-c pushed a commit that referenced this issue Jun 1, 2017

[cloud] Backport handling of CloudFormation int parameter values - fixes

748b7c4

#23593 (#25051)

ansibot added bug This issue/PR relates to a bug. and removed bug_report labels Mar 7, 2018

tedder mentioned this issue Aug 22, 2018

Cloudformation: The action DescribeStacks is not valid for this web service #44505

Closed

ansible locked and limited conversation to collaborators Apr 26, 2019

sivel unassigned ryansb Apr 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cloudformation module no longer works in 2.3.0 #23593

cloudformation module no longer works in 2.3.0 #23593

jagibson commented Apr 13, 2017

ansibot commented Apr 13, 2017

jagibson commented Apr 13, 2017

howinator commented Apr 14, 2017

jagibson commented Apr 14, 2017

tedder commented Apr 15, 2017 •

edited

s-hertel commented Apr 18, 2017

tedder commented Apr 18, 2017

jagibson commented Apr 21, 2017

mixja commented May 1, 2017

mixja commented May 1, 2017

s-hertel commented May 2, 2017

jagibson commented May 4, 2017 •

edited

tedder commented May 4, 2017

jagibson commented Jul 13, 2017

cloudformation module no longer works in 2.3.0 #23593

cloudformation module no longer works in 2.3.0 #23593

Comments

jagibson commented Apr 13, 2017

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

ansibot commented Apr 13, 2017

jagibson commented Apr 13, 2017

howinator commented Apr 14, 2017

jagibson commented Apr 14, 2017

tedder commented Apr 15, 2017 • edited

s-hertel commented Apr 18, 2017

tedder commented Apr 18, 2017

jagibson commented Apr 21, 2017

mixja commented May 1, 2017

mixja commented May 1, 2017

s-hertel commented May 2, 2017

jagibson commented May 4, 2017 • edited

tedder commented May 4, 2017

jagibson commented Jul 13, 2017

tedder commented Apr 15, 2017 •

edited

jagibson commented May 4, 2017 •

edited