Install fails with "the output has been hidden due to the fact that 'no_log: true' was specified for this result" #4111

sandrich · 2017-05-08T07:03:21Z

Description

When I try to install a single-node biy version I get the following error

TASK [openshift_metrics : slurp] ***********************************************
ok: [master.some_domiain.ch -> localhost]

TASK [openshift_metrics : generate htpasswd file for hawkular metrics] *********
fatal: [master.some_domiain.ch -> localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}

Version

ansible 2.2.2.0 (also tried with 2.3.0.0-3.el7)
openshift-ansible-3.5.67-1-4-g5b874ad

Steps To Reproduce

run ansible-playbook openshift-ansible/playbooks/byo/config.yml --inventory hosts

Expected Results

OpenShift to be installed correctly including metrics

ansible-playbook openshift-ansible/playbooks/byo/config.yml --inventory hosts
TASK [openshift_metrics : slurp] ***********************************************
ok: [master.some_domiain.ch -> localhost]

TASK [openshift_metrics : generate htpasswd file for hawkular metrics] *********
fatal: [master.some_domain.ch -> localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}

Additional Information

This is the inventory file I used

[OSEv3:children]
masters
nodes
etcd

[OSEv3:vars]
ansible_ssh_user=root
deployment_type=origin
openshift_node_kubelet_args={'max-pods': ['40'], 'image-gc-high-threshold': ['90'], 'image-gc-low-threshold': ['80']}
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
openshift_master_htpasswd_users={'chris': 'some_hash'}
openshift_master_default_subdomain=os.some_domain.ch

# Persistent Registry
openshift_hosted_registry_storage_kind=nfs
openshift_hosted_registry_storage_access_modes=['ReadWriteMany']
openshift_hosted_registry_storage_nfs_directory=/exports
openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)'
openshift_hosted_registry_storage_volume_name=registry
openshift_hosted_registry_storage_volume_size=10Gi

# Metrics
openshift_hosted_metrics_deploy=true
openshift_hosted_metrics_storage_kind=nfs
openshift_hosted_metrics_storage_access_modes=['ReadWriteOnce']
openshift_hosted_metrics_storage_nfs_directory=/exports
openshift_hosted_metrics_storage_nfs_options='*(rw,root_squash)'
openshift_hosted_metrics_storage_volume_name=metrics
openshift_hosted_metrics_storage_volume_size=5Gi
openshift_hosted_metrics_storage_host=master.some_domain.ch
openshift_hosted_metrics_public_url=https://hawkular-metrics.os.some_domain.ch/hawkular/metrics

# Logging
#openshift_hosted_logging_deploy=true
#openshift_hosted_logging_storage_kind=nfs
#openshift_hosted_logging_storage_access_modes=['ReadWriteOnce']
#openshift_hosted_logging_storage_nfs_directory=/exports
#openshift_hosted_logging_storage_nfs_options='*(rw,root_squash)'
#openshift_hosted_logging_storage_volume_name=logging
#openshift_hosted_logging_storage_volume_size=1Gi
#openshift_hosted_logging_hostname=logging.os.some_domain.ch
#openshift_hosted_logging_elasticsearch_cluster_size=1

[masters]
master.some_domain.ch openshift_ip=10.135.25.190 openshift_hostname=master openshift_public_ip=46.xxx.xxx.xxx openshift_public_hostname=master.some_domain.ch openshift_schedulable=True openshift_node_labels="{'region': 'infra'}"

[nodes]
master.some_domain.ch openshift_ip=10.135.25.190 openshift_hostname=master openshift_public_ip=46.xxx.xxx.xxx openshift_public_hostname=master.some_domain.ch

[etcd]
master.some_domain.ch openshift_ip=10.135.25.190 openshift_hostname=master openshift_public_ip=46.xxx.xxx.xxx openshift_public_hostname=master.some_domain.ch

[nfs]
master.some_domain.ch openshift_ip=10.135.25.190 openshift_hostname=master openshift_public_ip=46.xxx.xxx.xxx openshift_public_hostname=master.some_domain.ch

The text was updated successfully, but these errors were encountered:

sandrich · 2017-05-08T07:37:37Z

Nevermind. Fixed by downgrading ansible to 2.2.1

sdodson · 2017-05-08T12:57:36Z

re-opened as we need to fix anything that happens in 2.3 ASAP.

Can you verify though that you have python-passlib installed on your control host?

sandrich · 2017-05-08T13:23:28Z

I only got the following installed

[root@master ~]# rpm -qa | grep passlib
python2-passlib-1.6.5-1.el7.noarch

ewolinetz · 2017-05-09T16:01:46Z

@sdodson would it make sense to have the openshift_metrics role have an assert that the rpm exists on the control host before executing?

sdodson · 2017-05-09T19:10:48Z

yeah, but it looks like @un1x86 has it installed already so not sure why it's failing.

ewolinetz · 2017-05-09T20:35:10Z

@un1x86 just to verify, are you installing from your 'master' host, or is that your master node that you ran the rpm query from?

etsauer · 2017-05-10T00:03:28Z

@sdodson @ewolinetz getting the same result with openshift-ansible-3.6.61-1 and have python2-passlib-1.7.0-4.el7.noarch installed. Any other requirements?

sandrich · 2017-05-10T05:17:37Z

@ewolinetz I am installing it from the "master" host and I run the rpm query on the master host.

ewolinetz · 2017-05-10T15:32:30Z

I'm not sure if its the difference between python2-passlib and python-passlib, but on my control host I have python-passlib-1.6.2-2.el7.noarch installed and I don't see this problem.

sandrich · 2017-05-10T15:39:16Z

python2-passlib is automatically chosen when installing ansible from the openshift-origin repository.

ewolinetz · 2017-05-10T20:44:10Z

@un1x86 can you update roles/openshift_metrics/tasks/generate_hawkular_certificates.yaml to remove this line [1] and rerun your play with a more verbose output?

[1] https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_metrics/tasks/generate_hawkular_certificates.yaml#L27

etsauer · 2017-05-11T01:05:34Z

@ewolinetz I'm now on the same version of python-passlib as you are, and it seems to have gotten me past this issue. It turns out that versions of that rpm are available in multiple channels. In my case, it was coming from openstack-newton. I ran a yum remove python2-passlib and then ran yum install python-passlib --disablerepo openstack-newton and i was able to move forward with the install.

@un1x86 can you check yum info python2-passlib and see where you are getting it from? This was mine.

[root@93b2824028ac ~]# yum info python-passlib
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: mirror.beyondhosting.net
 * epel: mirror.math.princeton.edu
 * extras: distro.ibiblio.org
 * updates: chicago.gaminghost.co
Available Packages
Name        : python-passlib
Arch        : noarch
Version     : 1.6.2
Release     : 2.el7
Size        : 480 k
Repo        : epel/x86_64
Summary     : Comprehensive password hashing framework supporting over 20 schemes
URL         : http://passlib.googlecode.com
License     : BSD and Beerware and Copyright only
Description : Passlib is a password hashing library for Python 2 & 3, which provides
            : cross-platform implementations of over 20 password hashing algorithms,
            : as well as a framework for managing existing password hashes. It's
            : designed to be useful for a wide range of tasks, from verifying a hash
            : found in /etc/shadow, to providing full-strength password hashing for
            : multi-user application.

sandrich · 2017-05-11T06:04:11Z

@etsauer python-passlib is coming from epel while python2-passlib is coming from openshift-origin repository

Available Packages
Name        : python-passlib
Arch        : noarch
Version     : 1.6.2
Release     : 2.el7
Size        : 480 k
Repo        : epel/x86_64
Summary     : Comprehensive password hashing framework supporting over 20 schemes
URL         : http://passlib.googlecode.com
License     : BSD and Beerware and Copyright only
Description : Passlib is a password hashing library for Python 2 & 3, which provides
            : cross-platform implementations of over 20 password hashing algorithms,
            : as well as a framework for managing existing password hashes. It's
            : designed to be useful for a wide range of tasks, from verifying a hash
            : found in /etc/shadow, to providing full-strength password hashing for
            : multi-user application.

[root@master ~]# yum info python2-passlib
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.imt-systems.com
 * epel: mirror.imt-systems.com
 * extras: mirror.imt-systems.com
 * updates: mirror.imt-systems.com
Available Packages
Name        : python2-passlib
Arch        : noarch
Version     : 1.6.5
Release     : 1.el7
Size        : 488 k
Repo        : centos-openshift-origin
Summary     : Comprehensive password hashing framework supporting over 20 schemes
URL         : https://bitbucket.org/ecollins/passlib
License     : BSD and Beerware and Copyright only
Description : Passlib is a password hashing library for Python 2 & 3, which provides
            : cross-platform implementations of over 20 password hashing algorithms,
            : as well as a framework for managing existing password hashes. It's
            : designed to be useful for a wide range of tasks, from verifying a hash
            : found in /etc/shadow, to providing full-strength password hashing for
            : multi-user application.

@ewolinetz seems to run through like that. This is the output of the task.

TASK [openshift_metrics : generate htpasswd file for hawkular metrics] **************************************************************************************************************************************************************************************************
task path: /root/openshift-ansible/roles/openshift_metrics/tasks/generate_hawkular_certificates.yaml:45
 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: not {{ openshift_metrics_heapster_standalone | bool }}

Using module file /usr/lib/python2.7/site-packages/ansible/modules/web_infrastructure/htpasswd.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1494481580.2-8145002959404 `" && echo ansible-tmp-1494481580.2-8145002959404="` echo /root/.ansible/tmp/ansible-tmp-1494481580.2-8145002959404 `" ) && sleep 0'
<localhost> PUT /tmp/tmp3I55uB TO /root/.ansible/tmp/ansible-tmp-1494481580.2-8145002959404/htpasswd.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1494481580.2-8145002959404/ /root/.ansible/tmp/ansible-tmp-1494481580.2-8145002959404/htpasswd.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python2 /root/.ansible/tmp/ansible-tmp-1494481580.2-8145002959404/htpasswd.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1494481580.2-8145002959404/" > /dev/null 2>&1 && sleep 0'
changed: [master.some_domain.ch -> localhost] => {
    "changed": true,
    "invocation": {
        "module_args": {
            "attributes": null,
            "backup": null,
            "content": null,
            "create": true,
            "crypt_scheme": "apr_md5_crypt",
            "delimiter": null,
            "directory_mode": null,
            "follow": false,
            "force": null,
            "group": null,
            "mode": null,
            "name": "hawkular",
            "owner": null,
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "path": "/tmp/tmp.TWwJaE8eod/hawkular-metrics.htpasswd",
            "regexp": null,
            "remote_src": null,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "src": null,
            "state": "present",
            "unsafe_writes": null
        }
    },
    "msg": "Created /tmp/tmp.TWwJaE8eod/hawkular-metrics.htpasswd and added hawkular"
}

ewolinetz · 2017-05-11T13:49:33Z

@sdodson it seems that this may be something we need to change in our repo, or change how we gen the htpasswd file. Thoughts?

tonyturino · 2017-05-25T16:22:23Z

Having the same problem. I've tried everything including downgrading ansible to 2.2.1 and installing both flavors of passlib:

rpm -qa|grep passlib
python-passlib-1.6.2-2.el7.noarch
python2-passlib-1.6.5-1.el7.noarch

TASK [openshift_metrics : generate htpasswd file for hawkular metrics] *********
fatal: [origin-master.c.xxx-dev.internal -> localhost]: FAILED! => {"changed": false, "failed": true, "msg": "This module requires the passlib Python library"}

tonyturino · 2017-05-26T17:26:34Z

I'm still getting the message no matter if I've got python-passlib OR python2-passlib OR neither installed:

TASK [openshift_metrics : Check that python-passlib is available on the control host] **********************
fatal: [origin-master.c.mcomops-dev.internal]: FAILED! => {
"assertion": "'not installed' not in passlib_result.stdout",
"changed": false,
"evaluated_to": false,
"failed": true
}

MSG:

python-passlib rpm must be installed on control host

mjudeikis · 2017-05-30T10:53:50Z

Same error. removed no_logs from step.

TASK [openshift_metrics : generate htpasswd file for hawkular metrics] *********************************************************************
 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: not {{
openshift_metrics_heapster_standalone | bool }}

fatal: [852378-master1-dr.myriadpayments.com -> localhost]: FAILED! => {
    "changed": false, 
    "failed": true
}

MSG:

invalid version number '1.7.0.post20170223134505'

passlib:

[ocpadmin@806783-jbox1 share]$ rpm -qa|grep passlib
python2-passlib-1.7.0-4.el7.noarch

trying to rollback libs for this.
@sdodson Any ideas?

mjudeikis · 2017-05-30T11:00:51Z

downgrading passlib solved the issue.

[ocpadmin@806783-jbox1 ~]$ sudo yum search python2-passlib --showduplicates
Loaded plugins: product-id, search-disabled-repos, subscription-manager
======================================================= N/S matched: python2-passlib =======================================================
python2-passlib-1.6.5-1.el7.noarch : Comprehensive password hashing framework supporting over 20 schemes
python2-passlib-1.7.0-4.el7.noarch : Comprehensive password hashing framework supporting over 20 schemes

  Name and summary matches only, use "search all" for everything.
[ocpadmin@806783-jbox1 ~]$ sudo yum downgrade python2-passlib-1.6.5
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package python2-passlib.noarch 0:1.6.5-1.el7 will be a downgrade
---> Package python2-passlib.noarch 0:1.7.0-4.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================
 Package                          Arch                    Version                         Repository                                   Size
============================================================================================================================================
Downgrading:
 python2-passlib                  noarch                  1.6.5-1.el7                     rhel-7-server-ose-3.5-rpms                  488 k

Transaction Summary
============================================================================================================================================
Downgrade  1 Package

Total download size: 488 k
Is this ok [y/d/N]: y
Downloading packages:
python2-passlib-1.6.5-1.el7.noarch.rpm                                                                               | 488 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : python2-passlib-1.6.5-1.el7.noarch                                                                                       1/2 
  Cleanup    : python2-passlib-1.7.0-4.el7.noarch                                                                                       2/2 
  Verifying  : python2-passlib-1.6.5-1.el7.noarch                                                                                       1/2 
  Verifying  : python2-passlib-1.7.0-4.el7.noarch                                                                                       2/2 

Removed:
  python2-passlib.noarch 0:1.7.0-4.el7                                                                                                      

Installed:
  python2-passlib.noarch 0:1.6.5-1.el7

Seen same issue with metrics and logging deplpoyments/updates for 3.5 when using newest rpms

sdodson · 2017-05-30T12:29:13Z

Ansible pr to fix that ansible/ansible#20202 i'll see if we can get that pushed through. Which repo included python2-passlib-1.7.0?

mjudeikis · 2017-05-30T13:18:53Z

Looks like it wsa provided by Rackspace version of repos within spacewalk... raised with rackspace... :/

sferich888 · 2018-01-28T02:09:20Z

@sdodson and @mjudeikis I hit this #4111 (comment) on a 3.6 Origin install, when trying to add metrics.

# cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 

# yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: repos-va.psychz.net
 * epel: mirror.vcu.edu
 * extras: mirror.us.leaseweb.net
 * updates: mirror.vcu.edu
repo id                   repo name                                       status
base/7/x86_64             CentOS-7 - Base                                  9,591
centos-openshift-origin36 CentOS OpenShift Origin                          46+20
epel/x86_64               Extra Packages for Enterprise Linux 7 - x86_64  12,252
extras/7/x86_64           CentOS-7 - Extras                               313+16
updates/7/x86_64          CentOS-7 - Updates                               1,909
repolist: 24,111

# sudo yum search python2-passlib --showduplicates
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: repos-va.psychz.net
 * epel: mirror.vcu.edu
 * extras: mirror.us.leaseweb.net
 * updates: mirror.vcu.edu
========================= N/S matched: python2-passlib =========================
python2-passlib-1.7.0-4.el7.noarch : Comprehensive password hashing framework
                                   : supporting over 20 schemes
python2-passlib-1.7.0-4.el7.noarch : Comprehensive password hashing framework
                                   : supporting over 20 schemes

As it does not look like I have an option to downgrade, the issue seems unresolved.

sferich888 · 2018-01-28T02:21:33Z

If I manually delete the package, and pull https://centos.pkgs.org/7/centos-extras-x86_64/python-passlib-1.6.5-2.el7.noarch.rpm.html and install just this package (not sure why yum can't see this), I also can work around this.

sdodson · 2018-01-29T14:27:40Z

@sferich888 That's an ansible bug, are you running ansible 2.3.2 or 2.4.0 and newer?

ivar1234 · 2018-02-13T16:07:10Z

I got this fatal error upon running ansible roles using vault info on ansible 2.2.1.

fatal: [x.x.x.x]: FAILED! => {"ansible_facts": {}, "changed": false, "failed": true, "message": "Decryption failed on creds.yml"}

Going thru this above notes, I did tried putting passlib & upgrading ansible to 2.2.3 and getting the same result :( Cant find any other possible fix to this error. I can edit the vault file with the file location saved under vault-password-file on commandline and it works without fail. And while running the roles, I tried both options putting the ini parameter vault_password_file = ~ /.vault_file.txt in ansible.cfg and on the command line as --vault-password-file=~ /.vault_file.txt and neither works :( Appreciate a quick reply to this query.

-bash-4.1$ rpm -qa |grep passlib
python-passlib-1.5.3-1.el6.noarch
-bash-4.1$ ansible --version
ansible 2.2.3.0
config file = ~/ansible.cfg
configured module search path = ['./library']
-bash-4.1$ python --version
Python 2.6.6

Note: Intentionally gave space after ~ as its marking the end points to strike off the line in this markdown format.

sdodson · 2018-02-20T22:58:22Z

@ivar1234 I don't think we've ever tried to run openshift-ansible on a RHEL 6 control host. Our most commonly used platform would be RHEL/Centos 7 and current Fedora releases. Is it possible to run ansible on one of those?

0xmichalis · 2018-03-15T10:10:18Z

Seeing this issue when trying to install origin 3.9

PLAY [Determine openshift_version to configure on first master] ****************

TASK [Gathering Facts] *********************************************************
ok: [master1]

TASK [include_role] ************************************************************

TASK [openshift_version : Use openshift.common.version fact as version to configure if already installed] ***
ok: [master1]

TASK [openshift_version : include_tasks] ***************************************
included: /usr/share/ansible/openshift-ansible/roles/openshift_version/tasks/first_master_rpm_version.yml for master1

TASK [openshift_version : Set rpm version to configure if openshift_pkg_version specified] ***
skipping: [master1]

TASK [openshift_version : Set openshift_version for rpm installation] **********
included: /usr/share/ansible/openshift-ansible/roles/openshift_version/tasks/check_available_rpms.yml for master1

TASK [openshift_version : Get available origin version] ************************
ok: [master1]

TASK [openshift_version : fail] ************************************************
fatal: [master1]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
	to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.retry

[root@bastion ~]# ansible --version
ansible 2.4.2.0
  config file = /root/.ansible.cfg
  configured module search path = [u'/usr/share/ansible', u'/usr/share/ansible/openshift-ansible/library']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, May  3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)]

[root@master1 ~]# rpm -qa | grep passlib
python-passlib-1.6.5-2.el7.noarch

paulcalabro · 2018-10-22T16:27:06Z

Seeing this error as well:

fatal: [10.243.30.117]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

wyswust · 2019-03-27T08:59:03Z

how to solve when install ansible tower
An error occurred

fatal: [10.243.30.117]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

sandrich closed this as completed May 8, 2017

sdodson reopened this May 8, 2017

sdodson assigned ewolinetz May 8, 2017

sdodson added backport/3.3 kind/bug Categorizes issue or PR as related to a bug. priority/P1 and removed backport/3.3 labels May 8, 2017

ewolinetz mentioned this issue May 9, 2017

Adding assert to check for python-passlib on control host #4134

Merged

etsauer mentioned this issue May 16, 2017

Updating to OSP ocata repo, as there are some bugs with newton's channel redhat-cop/casl-ansible#44

Merged

ewolinetz mentioned this issue May 25, 2017

Updating python-passlib assert #4277

Merged

sdodson closed this as completed in #4277 May 26, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Install fails with "the output has been hidden due to the fact that 'no_log: true' was specified for this result" #4111

Install fails with "the output has been hidden due to the fact that 'no_log: true' was specified for this result" #4111

sandrich commented May 8, 2017

sandrich commented May 8, 2017

sdodson commented May 8, 2017

sandrich commented May 8, 2017

ewolinetz commented May 9, 2017

sdodson commented May 9, 2017

ewolinetz commented May 9, 2017

etsauer commented May 10, 2017

sandrich commented May 10, 2017

ewolinetz commented May 10, 2017

sandrich commented May 10, 2017

ewolinetz commented May 10, 2017

etsauer commented May 11, 2017 •

edited

sandrich commented May 11, 2017

ewolinetz commented May 11, 2017

tonyturino commented May 25, 2017

tonyturino commented May 26, 2017

mjudeikis commented May 30, 2017

mjudeikis commented May 30, 2017 •

edited

sdodson commented May 30, 2017

mjudeikis commented May 30, 2017

sferich888 commented Jan 28, 2018 •

edited

sferich888 commented Jan 28, 2018

sdodson commented Jan 29, 2018

ivar1234 commented Feb 13, 2018 •

edited

sdodson commented Feb 20, 2018

0xmichalis commented Mar 15, 2018 •

edited

paulcalabro commented Oct 22, 2018

wyswust commented Mar 27, 2019

Install fails with "the output has been hidden due to the fact that 'no_log: true' was specified for this result" #4111

Install fails with "the output has been hidden due to the fact that 'no_log: true' was specified for this result" #4111

Comments

sandrich commented May 8, 2017

Description

Version

Steps To Reproduce

Expected Results

Additional Information

sandrich commented May 8, 2017

sdodson commented May 8, 2017

sandrich commented May 8, 2017

ewolinetz commented May 9, 2017

sdodson commented May 9, 2017

ewolinetz commented May 9, 2017

etsauer commented May 10, 2017

sandrich commented May 10, 2017

ewolinetz commented May 10, 2017

sandrich commented May 10, 2017

ewolinetz commented May 10, 2017

etsauer commented May 11, 2017 • edited

sandrich commented May 11, 2017

ewolinetz commented May 11, 2017

tonyturino commented May 25, 2017

tonyturino commented May 26, 2017

mjudeikis commented May 30, 2017

mjudeikis commented May 30, 2017 • edited

sdodson commented May 30, 2017

mjudeikis commented May 30, 2017

sferich888 commented Jan 28, 2018 • edited

sferich888 commented Jan 28, 2018

sdodson commented Jan 29, 2018

ivar1234 commented Feb 13, 2018 • edited

sdodson commented Feb 20, 2018

0xmichalis commented Mar 15, 2018 • edited

paulcalabro commented Oct 22, 2018

wyswust commented Mar 27, 2019

etsauer commented May 11, 2017 •

edited

mjudeikis commented May 30, 2017 •

edited

sferich888 commented Jan 28, 2018 •

edited

ivar1234 commented Feb 13, 2018 •

edited

0xmichalis commented Mar 15, 2018 •

edited