Contribsys repo intermediary cert expired - bundle install fails with Could not verify the SSL certificate for https://gems.contribsys.com/ #5008

tcdachille · 2021-09-30T15:48:16Z

Similar to #4583, bundle install is failing with

Retrying fetcher due to error (2/6): Bundler::Fetcher::CertificateFailureError Could not verify the SSL certificate for https://gems.contribsys.com/.    
There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for verification. 
For information about OpenSSL certificates, see http://bit.ly/ruby-ssl. 
To connect without using SSL, edit your Gemfile sources and change 'https' to 'http'.    
/usr/lib64/ruby/2.6.0/net/protocol.rb:44: warning: exception in verify_callback is ignored

Trying https://www.ssllabs.com/ssltest/analyze.html?d=gems.contribsys.com reveals an expired cert

The text was updated successfully, but these errors were encountered:

mperham · 2021-09-30T15:58:54Z

The certificate seems to work with curl, Net::HTTP and Safari without issue.

> require 'net/https'
> Net::HTTP.get(URI("https://gems.contribsys.com"))
=> "This is Contributed Systems' Sidekiq Pro repo.\n"

mperham · 2021-09-30T16:02:34Z

https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

bpt2410 · 2021-09-30T16:03:32Z

We got the same issue

mperham · 2021-09-30T16:04:25Z

tl;dr: update ruby and openssl.

mperham · 2021-09-30T16:05:43Z

https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816

The behavior that causes this incompatibility was fixed over 4 years ago with the release of OpenSSL 1.1.0.

alejandrovelez7 · 2021-09-30T16:26:17Z

Updating bundler to >= 2.2 may be useful as well since older versions were using different ssl handling https://bundler.io/blog/2020/12/09/bundler-v2-2.html

LizhangX · 2021-09-30T16:43:50Z

Same here, using jruby docker image jruby:9.2.14.0-jdk.
Openssl is OpenSSL 1.1.0f 25 May 2017
But still getting the same issue during bundle install.

mperham · 2021-09-30T17:06:26Z

My current understanding: the core issue is that your system's trusted set of root certificates is years out of date. I can't update your system nor do I know how to do so because every system is different. The typical procedure is to update your operating system (e.g. Ubuntu 14.04 to 20.04) or OpenSSL package to get the latest certificate bundle included with it. On Debian/Ubuntu, see the ca-certificates package.

$ irb
> require 'net/http'
=> true
> Net::HTTP.get(URI("https://gems.contribsys.com"))
=> "This is Contributed Systems' Sidekiq Pro repo.\n"
> RUBY_VERSION
=> "2.7.0"
$ apt show ca-certificates
Package: ca-certificates
Version: 20210119~20.04.2

schmiddy · 2021-09-30T17:20:26Z

FYI we see this error on the latest Amazon Linux 2 AMI, you can check by launching an EC2 instance from public AMI ami-0c0c5692729d9c263 in us-west-2 (AMI Name amzn2-ami-kernel-5.10-hvm-2.0.20210813.1-x86_64-gp2).

[ec2-user@ip-172-31-40-244 ~]$ bundle --version
Bundler version 2.2.28

[ec2-user@ip-172-31-40-244 ~]$ ruby --version
ruby 2.6.8p205 (2021-07-07 revision 67951) [x86_64-linux]

Edit: I think the openssl version I posted at first was wrong, ignore that, it's probably "1.0.2k-fips"

mperham · 2021-09-30T17:24:54Z

https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190/214

davebenson · 2021-09-30T17:59:58Z

It appears to be a problem on google appengine's flex ruby environment as well. guess i'm SOL until google fixes it, unless i want to switch to a custom environment and build my own docker image. ugh.

dannyfallon · 2021-09-30T18:00:31Z

On Amazon Linux 2 we worked around this today by punting the expired certificate into the PKI blacklist on our machines:

trust dump --filter "pkcs11:id=%C4%A7%B1%A4%7B%2C%71%FA%DB%E1%4B%90%75%FF%C4%15%60%85%89%10;type=cert" > /etc/pki/ca-trust/source/blacklist/dst-root.p11-kit && update-ca-trust extract

I took the ID from trust list 👍

trust list | grep -A 5 -B 5 DST
pkcs11:id=%C4%A7%B1%A4%7B%2C%71%FA%DB%E1%4B%90%75%FF%C4%15%60%85%89%10;type=cert
    type: certificate
    label: DST Root CA X3
    trust: blacklisted
    category: authority

sm-kwohlhueter · 2021-09-30T19:39:24Z

We're using an AmazonLinux2 container for AWS CodeBuild, and @dannyfallon's one line'r fixed it for us.
🥇 for @dannyfallon

vishalzambre · 2021-09-30T21:09:43Z

We're using an AmazonLinux2 container for AWS CodeBuild, and @dannyfallon's one line'r fixed it for us.
🥇 for @dannyfallon

which one liner? I'm also using AmazonLinux2 with EBS, can you please help

sm-kwohlhueter · 2021-09-30T21:10:27Z

trust dump --filter "pkcs11:id=%C4%A7%B1%A4%7B%2C%71%FA%DB%E1%4B%90%75%FF%C4%15%60%85%89%10;type=cert" > /etc/pki/ca-trust/source/blacklist/dst-root.p11-kit && update-ca-trust extract

vishalzambre · 2021-09-30T21:12:17Z

It is not working on

NAME="Amazon Linux AMI"
VERSION="2018.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2018.03"
PRETTY_NAME="Amazon Linux AMI 2018.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

vishalzambre · 2021-09-30T21:12:26Z

Any other alternative?

sm-kwohlhueter · 2021-09-30T21:14:00Z

Try using AmazonLinux2. You seem to be on AmazonLinux.

vishalzambre · 2021-09-30T21:17:45Z

Any other workaround for AmazonLinux?

dmt219 · 2021-09-30T21:18:23Z

@vishalzambre This works for me on Amazon Linux 1:

Run trust dump --filter "pkcs11:id=%C4%A7%B1%A4%7B%2C%71%FA%DB%E1%4B%90%75%FF%C4%15%60%85%89%10;type=cert" in another Linux environment which supports the trust tool (Amazon Linux 1 doesn't have this)
In your Amazon Linux 1 instance, paste the result into
/etc/pki/ca-trust/source/blacklist/dst-root.p11-kit
Run update-ca-trust extract

vishalzambre · 2021-09-30T21:25:28Z

I'm getting now p11-kit: dst-root.p11-kit: nss-mozilla-ca-policy: invalid or unsupported attribute

vishalzambre · 2021-09-30T21:29:05Z

Now I can see

# update-ca-trust extract
p11-kit: dst-root.p11-kit: pkcs11: attribute before p11-kit section header
p11-kit: dst-root.p11-kit: pkcs11: attribute before p11-kit section header
p11-kit: dst-root.p11-kit: pkcs11: attribute before p11-kit section header
p11-kit: dst-root.p11-kit: pkcs11: attribute before p11-kit section header
p11-kit: dst-root.p11-kit: pkcs11: attribute before p11-kit section header

But still bundle install failing

Could not verify the SSL certificate for https://gems.contribsys.com/.
There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for verification. For information about OpenSSL certificates, see
http://bit.ly/ruby-ssl. To connect without using SSL, edit your Gemfile sources and change 'https' to 'http'.

sm-kwohlhueter · 2021-09-30T21:56:13Z

Is the ISRG Root X1 CA in your cert store?

On AmazonLinux2 it's at /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt. This could be different on AmazonLinux1.
When I run /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt | grep -i "ISRG Root X1" I get 1 result back.

vishalzambre · 2021-09-30T21:59:20Z

I can see

cat /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt  | grep -i "ISRG Root X1"
# ISRG Root X1

sm-kwohlhueter · 2021-09-30T22:10:13Z

That's good.
Looking at this thread there are 3 options:

find the AmazonLinux commands to blacklist the expired cert. (similar solution works on AmazonLinux2)
update to a newer version of OpenSSL. openssl version should be at least 1.1.0.
with the ISRG Root X1 CA Cert in your cert store, hopefully when certbot renews the *.contribsys.com certificate in a few days the expired CA cert will fall off the cert chain and this issue will go away.

dannyfallon · 2021-10-01T00:03:29Z

Not sure you have any interest in doing this Mike, but from reading this evening I think that you can support older OpenSSL versions (and things based on those), at the expense of Android >4, <7.1.1 support if you generate a new certificate using the --preferred-chain "ISRG Root X1" argument for Certbot. There's probably less Android users of Contribsys servers than folks using the likes of Amazon Linux or old OpenSSL versions 😅

bpt2410 · 2021-10-01T04:30:34Z

You can see this
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-expired-certificate/

vishalzambre · 2021-10-01T04:51:29Z

Finally below snippet worked for me

# cp -i /etc/pki/tls/certs/ca-bundle.crt ~/ca-bundle.crt-backup

# trust dump --filter "pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10" | openssl x509 | sudo tee /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem

# sudo update-ca-trust extract

vishalzambre · 2021-10-01T04:52:39Z

You can see this
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-expired-certificate/

Ahh it is really nice

adolfopeccin · 2021-10-01T06:59:26Z

Any solution for applications hosted on AWS Elastic Beanstalk?

bpt2410 · 2021-10-01T09:35:06Z

Any solution for applications hosted on AWS Elastic Beanstalk?

You can use this command on AWS EB
command: 'sudo sed -i"" "/Alias: DST Root CA X3/,/No Rejected Uses./d" /usr/share/pki/ca-trust-source/ca-bundle.trust.crt && sudo update-ca-trust'

tomchapin · 2021-10-01T19:12:50Z

If you are using Rails on AWS Elastic Beanstalk, you have to implement the ssl fix before the gems are installed.

I created a file named "/.ebextensions/000_ssl_fix.config" with these contents:

commands:
  create_pre_dir:
    command: "mkdir -p /opt/elasticbeanstalk/hooks/preinit"
    ignoreErrors: true

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/pre/00_fix_ssl.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash

      sudo sed -i"" "/Alias: DST Root CA X3/,/No Rejected Uses./d" /usr/share/pki/ca-trust-source/ca-bundle.trust.crt && sudo update-ca-trust

DaveCollinsJr · 2021-10-01T20:03:02Z

Having this same problem in Travis CI using "dist: xenial" we are working through some solutions to this and will post if we find something.

theist · 2021-10-01T20:53:28Z

I think I fixed it in an old xenial CI docker container by:

Updating to the last ca-certificates
Editing /etc/ca-certificates.conf
Placing a ! before line mozilla/DST_Root_CA_X3.crt
Running update-ca-certificates

After that I get this:

root@2d9486cf52f4:/app# ruby -v
ruby 2.3.3p222 (2016-11-21 revision 56859) [x86_64-linux]

root@2d9486cf52f4:/app# ruby
require 'net/https'
foo = Net::HTTP.get(URI("https://gems.contribsys.com"))
puts foo 
This is Contributed Systems' Sidekiq Pro repo.

root@2d9486cf52f4:/app# cat /etc/issue
Ubuntu 16.04.6 LTS \n \l

root@2d9486cf52f4:/app# openssl version
OpenSSL 1.0.2g  1 Mar 2016

But I haven't tried it yet for the real deal and there are a lot of moving parts when it comes to legacy systems, YMMV

dandynaufaldi · 2021-10-07T05:32:29Z

tried creating image with Jruby 9.2 and JDK 8 with base Ubuntu 20.04
then execute

require 'net/https'
Net::HTTP.get(URI('https://gems.contribsys.com'))

throws error

OpenSSL::SSL::SSLError: certificate verify failed
    connect_nonblock at org/jruby/ext/openssl/SSLSocket.java:255
  ssl_socket_connect at /opt/jruby/lib/ruby/stdlib/net/protocol.rb:44
             connect at /opt/jruby/lib/ruby/stdlib/net/http.rb:985
            do_start at /opt/jruby/lib/ruby/stdlib/net/http.rb:924
               start at /opt/jruby/lib/ruby/stdlib/net/http.rb:913
               start at /opt/jruby/lib/ruby/stdlib/net/http.rb:609
        get_response at /opt/jruby/lib/ruby/stdlib/net/http.rb:485
                 get at /opt/jruby/lib/ruby/stdlib/net/http.rb:462

looks like bug with jruby-openssl lib and this seems impacting everyone using jruby (not MRI)

related issues:

theist · 2021-10-07T06:45:51Z

Reviewing my own preliminary solution I can confirm this issue is solved in my CI system (several legacy MRI rubies in ubuntu 16.04) by just adding these steeps as root:

sed 's/mozilla.DST_Root_CA_X3.crt/!mozilla\/DST_Root_CA_X3.crt/' -i /etc/ca-certificates.conf
update-ca-certificates

Also worked in different combinations of debian / ubuntu systems and several MRI rubies.

richkettle · 2021-10-07T08:15:23Z

Having this same problem in Travis CI using "dist: xenial" we are working through some solutions to this and will post if we find something.

@DaveCollinsJr We are having the same issues on travis. I've been in contact with them and nothing they have suggested works. We previously used dist: trusty. But changing it and adding various lines of before_install haven't worked yet.

If anyone finds this looking for the travis issue what they suggested was:

dist: focal
before_install:
  - sudo apt-get update && sudo apt-get install apt-transport-https ca-certificates -y && sudo update-ca-certificates

or

dist: xenial
before_install:
  - sudo apt update && sudo apt install libgnutls-openssl27 libgnutls30

The above haven't worked for me as of yet. I'm currently stripping down our travis.yml to recreate with the minimum.

richkettle · 2021-10-07T09:47:02Z

Reviewing my own preliminary solution I can confirm this issue is solved in my CI system (several legacy MRI rubies in ubuntu 16.04) by just adding these steeps as root:
sed 's/mozilla.DST_Root_CA_X3.crt/!mozilla\/DST_Root_CA_X3.crt/' -i /etc/ca-certificates.conf
update-ca-certificates 
Also worked in different combinations of debian / ubuntu systems and several MRI rubies.

This worked for me on travis! Thank you!
I will forward it onto the travis team.

jordan-thoms · 2021-10-07T19:41:04Z

@dandynaufaldi We're also seeing this issue on JRuby, and the ca-certificates.conf workaround doesn't help there as it's using its own openssl implementation

mperham · 2021-10-07T20:26:01Z

Status update: the server cert has been renewed but still contains the expired root, as this is Let’s Encrypt’s choice. If the server is still not working for you, it is because your software is using an older version of OpenSSL which is incompatible.

igcherkaev · 2021-10-08T18:21:40Z

Status update: the server cert has been renewed but still contains the expired root, as this is Let’s Encrypt’s choice. If the server is still not working for you, it is because your software is using an older version of OpenSSL which is incompatible.

#5008 (comment) - if you follow this advice, and re-issue the cert for gems.contribsys.com the root CA certficiate will look like:

 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----

and most likely all problems for JRuby and others will end (except some android devices on certain versions, possibly).

mperham · 2021-10-08T18:26:36Z

@igcherkaev ubuntu 20.04 has certbot 0.40 which does not support the preferred chain option, else I would have done this. As is, I can't tell which of the two roots in fullchain.pem to exclude.

dbackeus · 2021-10-13T09:23:04Z

For the record: For those with issues on AWS Linux 2 I believe sudo yum update is enough to fix this issue at this point. Some updates to openssl and ca-certificates packages were released in the last week or so. At least I didn't personally have to run any other commands to fix SSL errors in our CI deployment.

davegallant · 2021-10-13T20:40:34Z

Major Updates: Update of ca-certificates to version 2021.2.50-72.amzn2.0.1 addresses the expiring IdentTrust DST Root CA X3, which affected some Let’s Encrypt TLS certificates. The effect of the expiring certificate would be an inability of OpenSSL to validate impacted certificates issued by Let’s Encrypt. Impacted customers may have experienced connection or certificate errors when attempting to connect to certain websites or APIs that use Let's Encrypt certificates.

https://aws.amazon.com/amazon-linux-2/release-notes/

brendanstennett · 2021-10-15T19:04:38Z

Running on a docker base image of openjdk:8-jdk-buster as of today is resulting in the same issue. Since this seems to be relevant to the certificate choice of gems.contribsys.com, which is used for a paid version of sidekiq, would you consider using a SSL certificate from a different trusted authority while we wait for this to settle?

mperham · 2021-10-15T19:35:54Z

@brendanstennett I have a potential fix which I will roll out later tonight after US business hours. The fix should last until early December, when the cert renews again.

mperham · 2021-10-15T23:03:44Z

I've removed the old, expired root reference. jruby-openssl should work now.

KLForsythe · 2021-12-06T17:06:33Z

@mperham You mentioned that the fix implemented in #5008 (comment) should last until early December. We are using sidekiq-pro, and our builds are newly failing due to this issue Could not verify the SSL certificate for https://gems.contribsys.com/ Can you implement the fix for your renewed certs?

mperham · 2021-12-06T17:16:31Z

@KLForsythe The cert has not changed since Oct 15th and no one else is reporting a problem. I'm giving one quarter for people to roll out any necessary client upgrades (LE certs renew every quarter). The current cert expires on Jan 5 2022 so it will be renewed very soon.

j3ck · 2022-02-03T22:40:49Z

mac os 12.1

sed 's/mozilla.DST_Root_CA_X3.crt/!mozilla\/DST_Root_CA_X3.crt/' -i /etc/ssl/cert.pem
SSL_CERT_FILE=/etc/ssl/cert.pem bundle install

to make it work without ENV variable, with RVM

rvm osx-ssl-certs update all # will update cert.pem in all places
bundle install

upd:
for ruby 2.3.8 the only way to fix is reinstall ruby with openssl 1.0.2(with versions below only SSL_CERT_FILE helps)

rvm reinstall 2.3.8 --with-openssl-dir=/usr/local/opt/openssl@1.0.2t

davegallant mentioned this issue Oct 1, 2021

Update ca-certificates to remove DST Root CA X3 rewindio/docker-amazonlinux2-ruby#9

Merged

stoivo mentioned this issue Oct 1, 2021

Expired certificate? idank/explainshell#287

Closed

mperham closed this as completed Oct 22, 2021

mperham mentioned this issue Jan 26, 2022

[Question] OpenSSL certificate error in sidekiq gem installing #5144

Closed

mperham mentioned this issue Sep 1, 2023

Issue with SSL certificate when I try to install sidekiq-pro gem #6019

Closed

Contribsys repo intermediary cert expired - bundle install fails with Could not verify the SSL certificate for https://gems.contribsys.com/ #5008

Contribsys repo intermediary cert expired - bundle install fails with Could not verify the SSL certificate for https://gems.contribsys.com/ #5008

Comments

tcdachille commented Sep 30, 2021 • edited Loading

mperham commented Sep 30, 2021

mperham commented Sep 30, 2021

bpt2410 commented Sep 30, 2021

mperham commented Sep 30, 2021

mperham commented Sep 30, 2021

alejandrovelez7 commented Sep 30, 2021

LizhangX commented Sep 30, 2021

mperham commented Sep 30, 2021

schmiddy commented Sep 30, 2021 • edited Loading

mperham commented Sep 30, 2021

davebenson commented Sep 30, 2021

dannyfallon commented Sep 30, 2021 • edited Loading

sm-kwohlhueter commented Sep 30, 2021

vishalzambre commented Sep 30, 2021

sm-kwohlhueter commented Sep 30, 2021

vishalzambre commented Sep 30, 2021

vishalzambre commented Sep 30, 2021

sm-kwohlhueter commented Sep 30, 2021

vishalzambre commented Sep 30, 2021

dmt219 commented Sep 30, 2021 • edited Loading

vishalzambre commented Sep 30, 2021

vishalzambre commented Sep 30, 2021

sm-kwohlhueter commented Sep 30, 2021

vishalzambre commented Sep 30, 2021

sm-kwohlhueter commented Sep 30, 2021

dannyfallon commented Oct 1, 2021

bpt2410 commented Oct 1, 2021

vishalzambre commented Oct 1, 2021

vishalzambre commented Oct 1, 2021

adolfopeccin commented Oct 1, 2021

bpt2410 commented Oct 1, 2021

tomchapin commented Oct 1, 2021

DaveCollinsJr commented Oct 1, 2021

theist commented Oct 1, 2021

dandynaufaldi commented Oct 7, 2021

theist commented Oct 7, 2021 • edited Loading

richkettle commented Oct 7, 2021

richkettle commented Oct 7, 2021 • edited Loading

jordan-thoms commented Oct 7, 2021

mperham commented Oct 7, 2021

igcherkaev commented Oct 8, 2021

mperham commented Oct 8, 2021 • edited Loading

dbackeus commented Oct 13, 2021

davegallant commented Oct 13, 2021

brendanstennett commented Oct 15, 2021 • edited Loading

mperham commented Oct 15, 2021

mperham commented Oct 15, 2021

KLForsythe commented Dec 6, 2021

mperham commented Dec 6, 2021

j3ck commented Feb 3, 2022 • edited Loading

tcdachille commented Sep 30, 2021 •

edited

Loading

schmiddy commented Sep 30, 2021 •

edited

Loading

dannyfallon commented Sep 30, 2021 •

edited

Loading

dmt219 commented Sep 30, 2021 •

edited

Loading

theist commented Oct 7, 2021 •

edited

Loading

richkettle commented Oct 7, 2021 •

edited

Loading

mperham commented Oct 8, 2021 •

edited

Loading

brendanstennett commented Oct 15, 2021 •

edited

Loading

j3ck commented Feb 3, 2022 •

edited

Loading