Minimum permissions required for use of Amazon cloud modules, and improve ansible error output #22552

ghost · 2017-03-13T03:17:26Z

Documentation Report

COMPONENT NAME

aws modules (various)

ANSIBLE VERSION

2.2.1.0

Documentation for AWS specific modules (ec2_vol etc), should outline which IAM permissions are required for the module to work.

EXPECTED RESULTS

There should be a note on the AWS specific module outlining the list of IAM permissions that module uses to do it's job. Otherwise the end user is left with the prospect of pouring through ansible source code, and trying to guess what has been called. At the very least, the sts coded errors should be displayed to the user in the output together with the existing 'permission denied' error.

ACTUAL RESULTS

Currently the error message of 'permission denied' is pretty useless, as you don't know WHICH permission was denied, or which was being attempted, either.

s-hertel · 2017-03-16T12:01:52Z

I've been looking into this, and it would 1) take a long time 2) need to be updated regularly. Some modules will be more messy to document than others. As I am unable to focus solely on this, I'd welcome anyone interested to submit PRs. It is a documentation bug and I feel like fixing it would be very helpful (as I've tripped over this problem a few times myself). I will start requesting for new modules to document this.

ghost · 2017-03-16T18:55:09Z

i did a quick and dirty scan with:
for file in lib/ansible/modules/cloud/amazon/*.py; do echo -e "\nFILE: $file"; sed 's/.*ec2./ec2./g' $file | cut -d ' ' -f1 | sort | grep ec2 | cut -d '(' -f1 | uniq | grep -v ec2_; done

output of which is below:

but yeah, it would take some upkeep. I think beyond just the documentation that more helpful error messages from ansible would be good. I don't know if boto is the weak link there, but the errors messages some aws tools give where they contain the sts encoded error message would be great (the ones you can then decode with aws sts decode-authorization-message --encoded-message) because at least then you know which permission was missing.

FILE: lib/ansible/modules/cloud/amazon/aws_kms.py
ec2.boto3_conn
ec2.get_aws_connection_info
FILE: lib/ansible/modules/cloud/amazon/cloudformation_facts.py
FILE: lib/ansible/modules/cloud/amazon/cloudformation.py
ec2.ansible_dict_to_boto3_tag_list
ec2.boto3_conn
ec2.get_aws_connection_info
FILE: lib/ansible/modules/cloud/amazon/cloudfront_facts.py
FILE: lib/ansible/modules/cloud/amazon/cloudtrail.py
FILE: lib/ansible/modules/cloud/amazon/cloudwatchevent_rule.py
FILE: lib/ansible/modules/cloud/amazon/dynamodb_table.py
FILE: lib/ansible/modules/cloud/amazon/ec2_ami_copy.py
ec2.copy_image
ec2.create_tags
ec2.get_waiter
FILE: lib/ansible/modules/cloud/amazon/ec2_ami_find.py
ec2.blockdevicemapping
ec2.get_all_images
FILE: lib/ansible/modules/cloud/amazon/ec2_ami.py
ec2.blockdevicemapping
ec2.create_image
ec2.create_tags
ec2.delete_snapshot
ec2.deregister_image
ec2.get_all_images
ec2.get_image
ec2.register_image
FILE: lib/ansible/modules/cloud/amazon/_ec2_ami_search.py
FILE: lib/ansible/modules/cloud/amazon/ec2_asg_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_asg.py
ec2.autoscale
ec2.autoscale,
ec2.elb,
#log.basicConfig
FILE: lib/ansible/modules/cloud/amazon/ec2_customer_gateway.py
ec2.create_customer_gateway
ec2.create_tags
ec2.delete_customer_gateway
ec2.describe_customer_gateways
FILE: lib/ansible/modules/cloud/amazon/ec2_eip.py
ec2.allocate_address
ec2.associate_address
ec2.disassociate_address
ec2.get_all_addresses
ec2.get_all_network_interfaces
ec2.get_all_reservations
ec2.instance_ids
FILE: lib/ansible/modules/cloud/amazon/ec2_elb_facts.py
ec2.elb
ec2.elb,
ec2.tag
FILE: lib/ansible/modules/cloud/amazon/ec2_elb_lb.py
ec2.elb
ec2.elb,
ec2.elb.attributes
ec2.elb.attributes.LbAttributes
ec2.elb.healthcheck
ec2.get_all_security_groups
ec2.tag
FILE: lib/ansible/modules/cloud/amazon/ec2_elb.py
ec2.autoscale
ec2.autoscale,
ec2.elb
ec2.elb,
ec2.get_only_instances
ec2.InstanceObject
FILE: lib/ansible/modules/cloud/amazon/ec2_eni_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_eni.py
FILE: lib/ansible/modules/cloud/amazon/ec2_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_group_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_group.py
ec2.authorize_security_group_egress
ec2.create_security_group
ec2.get_all_instances
ec2.get_all_security_groups
ec2.revoke_security_group_egress
ec2.securitygroup
ec2.securitygroup.IPPermissions
FILE: lib/ansible/modules/cloud/amazon/ec2_key.py
ec2.create_key_pair
ec2.get_key_pair
ec2.import_key_pair
FILE: lib/ansible/modules/cloud/amazon/ec2_lc_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_lc_find.py
FILE: lib/ansible/modules/cloud/amazon/ec2_lc.py
ec2.autoscale
ec2.autoscale,
ec2.blockdevicemapping
FILE: lib/ansible/modules/cloud/amazon/ec2_metric_alarm.py
ec2.cloudwatch
ec2.cloudwatch,
FILE: lib/ansible/modules/cloud/amazon/ec2.py
ec2.blockdevicemapping
ec2.create_tags
ec2.get_all_instances
ec2.get_all_security_groups
ec2.get_all_snapshots
ec2.get_all_spot_instance_requests
ec2.instance_ids
ec2.instances
ec2.modify_network_interface_attribute
ec2.networkinterface.NetworkInterfaceCollection
ec2.networkinterface.NetworkInterfaceSpecification
ec2.request_spot_instances
ec2.run_instances
ec2.terminate_instances
FILE: lib/ansible/modules/cloud/amazon/ec2_remote_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_scaling_policy.py
ec2.autoscale
ec2.autoscale,
FILE: lib/ansible/modules/cloud/amazon/ec2_snapshot_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_snapshot.py
ec2.create_snapshot
ec2.delete_snapshot
ec2.get_all_snapshots
ec2.get_all_volumes
FILE: lib/ansible/modules/cloud/amazon/ec2_tag.py
ec2.create_tags
ec2.delete_tags
ec2.get_all_tags
ec2.tagged_instances
FILE: lib/ansible/modules/cloud/amazon/ec2_vol_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vol.py
ec2.blockdevicemapping
ec2.create_tags
ec2.create_volume
ec2.delete_volume
ec2.get_all_instances
ec2.get_all_volumes
ec2.get_password_data
ec2.instances
ec2.modify_instance_attribute
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_dhcp_options_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_dhcp_options.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_igw_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_igw.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_nacl_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_nacl.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_nat_gateway_facts.py
ec2.html#EC2.Client.describe_nat_gateways
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_nat_gateway.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_net_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_net.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_peer.py
FILE: lib/ansible/modules/cloud/amazon/_ec2_vpc.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_route_table_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_route_table.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_subnet_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_subnet.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_vgw_facts.py
FILE: lib/ansible/modules/cloud/amazon/ec2_vpc_vgw.py
FILE: lib/ansible/modules/cloud/amazon/ec2_win_password.py
ec2.get_password_data
FILE: lib/ansible/modules/cloud/amazon/ecs_cluster.py
FILE: lib/ansible/modules/cloud/amazon/ecs_ecr.py
FILE: lib/ansible/modules/cloud/amazon/ecs_service_facts.py
FILE: lib/ansible/modules/cloud/amazon/ecs_service.py
FILE: lib/ansible/modules/cloud/amazon/ecs_taskdefinition.py
FILE: lib/ansible/modules/cloud/amazon/ecs_task.py
FILE: lib/ansible/modules/cloud/amazon/efs_facts.py
FILE: lib/ansible/modules/cloud/amazon/efs.py
FILE: lib/ansible/modules/cloud/amazon/elasticache_parameter_group.py
FILE: lib/ansible/modules/cloud/amazon/elasticache.py
FILE: lib/ansible/modules/cloud/amazon/elasticache_snapshot.py
FILE: lib/ansible/modules/cloud/amazon/elasticache_subnet_group.py
FILE: lib/ansible/modules/cloud/amazon/execute_lambda.py
FILE: lib/ansible/modules/cloud/amazon/iam_cert.py
FILE: lib/ansible/modules/cloud/amazon/iam_mfa_device_facts.py
FILE: lib/ansible/modules/cloud/amazon/iam_policy.py
FILE: lib/ansible/modules/cloud/amazon/iam.py
FILE: lib/ansible/modules/cloud/amazon/iam_role.py
ec2.amazonaws.com'
FILE: lib/ansible/modules/cloud/amazon/iam_server_certificate_facts.py
FILE: lib/ansible/modules/cloud/amazon/init.py
FILE: lib/ansible/modules/cloud/amazon/kinesis_stream.py
FILE: lib/ansible/modules/cloud/amazon/lambda_alias.py
FILE: lib/ansible/modules/cloud/amazon/lambda_event.py
FILE: lib/ansible/modules/cloud/amazon/lambda_facts.py
FILE: lib/ansible/modules/cloud/amazon/lambda.py
FILE: lib/ansible/modules/cloud/amazon/rds_param_group.py
FILE: lib/ansible/modules/cloud/amazon/rds.py
FILE: lib/ansible/modules/cloud/amazon/rds_subnet_group.py
FILE: lib/ansible/modules/cloud/amazon/redshift.py
FILE: lib/ansible/modules/cloud/amazon/redshift_subnet_group.py
FILE: lib/ansible/modules/cloud/amazon/route53_facts.py
FILE: lib/ansible/modules/cloud/amazon/route53_health_check.py
FILE: lib/ansible/modules/cloud/amazon/route53.py
FILE: lib/ansible/modules/cloud/amazon/route53_zone.py
FILE: lib/ansible/modules/cloud/amazon/s3_bucket.py
FILE: lib/ansible/modules/cloud/amazon/s3_lifecycle.py
FILE: lib/ansible/modules/cloud/amazon/s3_logging.py
FILE: lib/ansible/modules/cloud/amazon/s3.py
FILE: lib/ansible/modules/cloud/amazon/s3_sync.py
ec2.boto3_conn
ec2.get_aws_connection_info
FILE: lib/ansible/modules/cloud/amazon/s3_website.py
FILE: lib/ansible/modules/cloud/amazon/sns_topic.py
FILE: lib/ansible/modules/cloud/amazon/sqs_queue.py
FILE: lib/ansible/modules/cloud/amazon/sts_assume_role.py
FILE: lib/ansible/modules/cloud/amazon/sts_session_token.py

michael-dev2rights · 2017-03-26T10:19:59Z

It is very difficult to fully document the requirements of each module because those requirements will vary depending on how the module is used. For example, a module that is only used in "state: present" mode will need only create rights and should not have delete rights, whilst if you use the same module with "state: absent" you will need delete rights.

My proposal is to

start with an example policy based on the one I am using for integration testing
try to extend the integration tests so that they exercise every single case possible
systematically improve the error messages as we find cases where they aren't good enough.

@robertpearce I've got an active pull request which is fixing the integration tests and improving the error messages as I go along (see #22499) could you please merge that into your ansible and see if it improves your error messages. Please, then list each case where you get a useless error message and we can try to fix it.

When you find a case, please report the YAML for the task which caused the error and the detailed output of running ansible-playbook -vvv ; please @ me in the report (it can just be a comment in this ticket as far as I'm concerned.

wimnat · 2017-05-14T11:52:58Z

The newer boto3 modules provide better errors. For example, testing some newer modules you get the following:

User: arn:aws:iam::111111111111:user/rob.white is not authorized to perform: iam:GetRole on resource: role testrole

So it's pretty clear what IAM permission i need to grant.

ikarlashov · 2017-05-24T08:02:36Z

@wimnat I can't confirm. I'm trying to do a simple task: start instance with ansible, and it's a big headache for me (even for AWS support). I assigned EC2 full access for IAM, simulated a policy, confirmed with AWS support, but ansible didn't work. And I don't see the detailed error. Only:

UnauthorizedOperationYou are not authorized to perform this operation.

So yes, all essential modules should be served with the list of approximate permissions.

PS. It's rediculous when u have ec2 full access role and you can't even start instance.

wimnat · 2017-05-24T08:44:21Z

Can you show me one of your AWS tasks?

Have you tried using AWS cli on the same instance? Does that work?

ikarlashov · 2017-05-24T11:31:51Z

@wimnat Yes, I've just started the instance from aws cli:

{ "StartingInstances": [ { "InstanceId": "i-0521c48cae9d66145", "CurrentState": { "Code": 0, "Name": "pending" }, "PreviousState": { "Code": 80, "Name": "stopped" } } ] }

From ansible - still no luck. Playbook:

---
- hosts: localhost
  gather_facts: False
  connection: local
  vars:
    aws_access_key: 'ACCESS_KEY_CODE'
    aws_secret_key: 'SECRET_KEY_CODE'
    instance_name: 'NLP-EN'
    aws_region: 'eu-central-1'

  tasks:
    - local_action:
        module: ec2
        instance_tags:
          Name: '{{ instance_name }}'
        region: '{{ aws_region }}'
        state: running
        aws_access_key: '{{aws_access_key}}'
        aws_secret_key: '{{aws_secret_key}}'

I tried weeks ago with the same playbook - all worked fine. Now it doesn't work.

boto==2.46.1
boto3==1.4.4
botocore==1.5.53
ansible 2.3

wimnat · 2017-05-24T11:55:21Z

When you said in your previous post that you had 'assigned EC2 full access for IAM' what do you mean exactly? Have you assigned an iam role to the instance? If yes, then the aws credentials in your task are unnecessary.

ikarlashov · 2017-05-24T12:19:42Z

I assigned EC2 full access for the whole ec2 service within eu-central-1 region. Instance ID is asterisk.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "*",
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:eu-central-1:888728039934:*"
        }
    ]
}

Why do you think credentials are unnecessary? I didn't configure aws cli profile on my machine(only for tests). And ansible doesn't know my AWS creds.

wimnat · 2017-05-24T12:46:47Z

Ansible doesn't need to know your creds if you're using iam roles. Thats the whole point in them - to avoid having to use access keys on EC2 instances. However, iam service role are relatively new and I'm not sure on the boto3 or ansible support. If you just create an instance role and assign it to the ansible instance you should be fine.

wimnat · 2017-05-24T12:48:32Z

Also, I would seriously consider modifying that policy of only for testing. If any one got access to one of your instances they could perform absolutely any action

ikarlashov · 2017-05-24T13:36:34Z

@wimnat that's the point. I used 3 actions:StartInstances, StopInstances, DescribeInstances and succedded with instance start/stop. Now, even with EC2 full previliges I'm unable to start instance.

You thought I'm using ansible on EC2 machine, but no, It's my local machine. That's why I have to specify aws creds in playbook :)

wimnat · 2017-05-24T14:13:11Z

Ok I'm confused. I thought your play wasn't working at all.

So you ran it first time and it worked and then you rerun it and got an author error?

Check the iam permissions of your user. As you're running from your local machine, iam service roles are not going to make any difference

ikarlashov · 2017-05-24T14:22:36Z

It worked a few weeks ago. Permissions are ok, because I used this creds in aws cli to run instance. That's why ansible+AWS IAM is very weeeeired.

wimnat · 2017-05-24T14:24:00Z

So what's the exact error?

ikarlashov · 2017-05-24T14:27:06Z

Error I mentioned in my first post:

UnauthorizedOperationYou are not authorized to perform this operation.

detailed: https://pastebin.com/nNmGbY6g

wimnat · 2017-05-24T14:34:16Z

Well then it must be your user permissions. What's your policy look like? Try addingfull EC2 permissions or turn on CloudTrail and see what calls are failing The module could very well behaved differently on subsequent runs

…

On 25 May 2017 12:27 am, "klausitto" ***@***.***> wrote: Error I mentioned in my first post: UnauthorizedOperationYou are not authorized to perform this operation. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22552 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHFORhxoSTPlpG0fH_aiZ7pUpzG2aNSmks5r9D5ngaJpZM4Mazbe> .

ikarlashov · 2017-05-24T14:37:46Z

I pasted my policy a few messages above :)

s-hertel · 2017-05-24T21:20:56Z

@klausitto Yeah, so you've hit upon the crux of the problem. Different runs will use different permissions (and determining what permissions you may require means you have to trace both the ansible code and boto code) and since the ec2 module uses boto2 as opposed to boto3 it doesn't provide helpful responses when it gets a 403 Forbidden (permissions) error. Your policy+playbook combination doesn't work for me either. It appears that resources beyond your policy are being used. I have a hunch it has to do with boto trying to do a DescribeInstances action and since you're only allowing actions to be done to a particular instance that is causing the issue. Changes resources to "*" works for me. It's just hard to determine what the issue is. This problem will hopefully be simpler to fix once things are ported to boto3. I'm sorry you're dealing with this.

ikarlashov · 2017-05-25T07:18:52Z

@s-hertel If you look at my policy, I'm not restricting access for particular instances. I've provided a full access to entire region.

UPD: Hint regarding CloudTrail was very helpful. Thanks @wimnat. Final IAM policy to run Instance:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "ec2:StartInstances",
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:eu-central-1:888728039934:instance/i-0521c48cae9d55145"
        },
        {
            "Effect": "Allow",
            "Action": "ec2:Describe*",
            "Resource": "*"
        }
    ]
}

s-hertel · 2017-05-25T17:27:12Z

@klausitto Yes. :)

ansibot · 2017-09-09T22:23:51Z

cc @dharmabumstead
click here for bot help

ansibot · 2017-11-21T22:24:09Z

cc @willthames
click here for bot help

ansibot · 2020-02-01T00:03:13Z

cc @jillr @tremble
click here for bot help

ansibot · 2020-08-17T14:48:02Z

Thank you very much for your interest in Ansible. Ansible has migrated much of the content into separate repositories to allow for more rapid, independent development. We are closing this issue/PR because this content has been moved to one or more collection repositories.

lib/ansible/modules/cloud/amazon -> https://galaxy.ansible.com/community/aws

For further information, please see:
https://github.com/ansible/ansibullbot/blob/master/docs/collection_migration.md

ansibot added affects_2.2 This issue/PR affects Ansible v2.2 aws cloud docs_report module This issue/PR relates to a module. needs_triage Needs a first human triage before being processed. labels Mar 13, 2017

s-hertel self-assigned this Mar 13, 2017

s-hertel removed the needs_triage Needs a first human triage before being processed. label Mar 13, 2017

s-hertel mentioned this issue Mar 14, 2017

S3 module documentation should cover minimal amount of S3 actions needed to function #22622

Closed

s-hertel changed the title ~~document aws module IAM permission requirements, and improve ansible error output~~ Minimum permissions required for use of Amazon cloud modules, and improve ansible error output Apr 19, 2017

s-hertel mentioned this issue Apr 19, 2017

Minimum privaleges required for use of Amazon cloud modules #23704

Closed

ansibot removed the module This issue/PR relates to a module. label Apr 19, 2017

ansibot added the support:core This issue/PR relates to code supported by the Ansible Engineering Team. label Jun 29, 2017

shitalmule04 added a commit to shitalmule04/ansible that referenced this issue Nov 4, 2017

Bug fixed Amazon cloud module ansible#22552

5047acb

shitalmule04 mentioned this issue Nov 4, 2017

Bug fixed issue #22552 #32566

Closed

julienvey mentioned this issue Feb 21, 2018

ec2_asg_facts throws errors with AutoScalingReadOnly Policy #36192

Closed

ansibot added docs This issue/PR relates to or includes documentation. and removed docs_report labels Mar 1, 2018

s-hertel mentioned this issue Nov 29, 2018

parse botocore.endpoint logs into a list of AWS actions #49312

Merged

ansibot added the has_pr This issue has an associated PR. label Jul 28, 2019

ansibot removed the needs_maintainer Ansibot is unable to identify maintainers for this PR. (Check `author` in docs or BOTMETA.yml) label Feb 1, 2020

ansibot added collection Related to Ansible Collections work collection:community.general needs_collection_redirect https://github.com/ansible/ansibullbot/blob/master/docs/collection_migration.md labels Apr 29, 2020

ansibot added collection:community.aws and removed collection:community.general labels Jun 14, 2020

ansibot added the bot_closed label Aug 17, 2020

ansibot closed this as completed Aug 17, 2020

ansible locked and limited conversation to collaborators Sep 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minimum permissions required for use of Amazon cloud modules, and improve ansible error output #22552

Minimum permissions required for use of Amazon cloud modules, and improve ansible error output #22552

ghost commented Mar 13, 2017

s-hertel commented Mar 16, 2017 •

edited

Loading

ghost commented Mar 16, 2017 •

edited by ghost

Loading

michael-dev2rights commented Mar 26, 2017 •

edited

Loading

wimnat commented May 14, 2017

ikarlashov commented May 24, 2017 •

edited

Loading

wimnat commented May 24, 2017

ikarlashov commented May 24, 2017 •

edited

Loading

wimnat commented May 24, 2017

ikarlashov commented May 24, 2017

wimnat commented May 24, 2017

wimnat commented May 24, 2017

ikarlashov commented May 24, 2017

wimnat commented May 24, 2017

ikarlashov commented May 24, 2017

wimnat commented May 24, 2017

ikarlashov commented May 24, 2017 •

edited

Loading

wimnat commented May 24, 2017 via email

ikarlashov commented May 24, 2017

s-hertel commented May 24, 2017 •

edited

Loading

ikarlashov commented May 25, 2017 •

edited

Loading

s-hertel commented May 25, 2017

ansibot commented Sep 9, 2017

ansibot commented Nov 21, 2017

ansibot commented Feb 1, 2020

ansibot commented Aug 17, 2020

Minimum permissions required for use of Amazon cloud modules, and improve ansible error output #22552

Minimum permissions required for use of Amazon cloud modules, and improve ansible error output #22552

Comments

ghost commented Mar 13, 2017

COMPONENT NAME

ANSIBLE VERSION

EXPECTED RESULTS

ACTUAL RESULTS

s-hertel commented Mar 16, 2017 • edited Loading

ghost commented Mar 16, 2017 • edited by ghost Loading

michael-dev2rights commented Mar 26, 2017 • edited Loading

wimnat commented May 14, 2017

ikarlashov commented May 24, 2017 • edited Loading

wimnat commented May 24, 2017

ikarlashov commented May 24, 2017 • edited Loading

wimnat commented May 24, 2017

ikarlashov commented May 24, 2017

wimnat commented May 24, 2017

wimnat commented May 24, 2017

ikarlashov commented May 24, 2017

wimnat commented May 24, 2017

ikarlashov commented May 24, 2017

wimnat commented May 24, 2017

ikarlashov commented May 24, 2017 • edited Loading

wimnat commented May 24, 2017 via email

ikarlashov commented May 24, 2017

s-hertel commented May 24, 2017 • edited Loading

ikarlashov commented May 25, 2017 • edited Loading

s-hertel commented May 25, 2017

ansibot commented Sep 9, 2017

ansibot commented Nov 21, 2017

ansibot commented Feb 1, 2020

ansibot commented Aug 17, 2020

s-hertel commented Mar 16, 2017 •

edited

Loading

ghost commented Mar 16, 2017 •

edited by ghost

Loading

michael-dev2rights commented Mar 26, 2017 •

edited

Loading

ikarlashov commented May 24, 2017 •

edited

Loading

ikarlashov commented May 24, 2017 •

edited

Loading

ikarlashov commented May 24, 2017 •

edited

Loading

s-hertel commented May 24, 2017 •

edited

Loading

ikarlashov commented May 25, 2017 •

edited

Loading