Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker-credential-gcloud + gcloud 298.0.0 + ubuntu-18.04: "OPENSSL_1_1_1 not found" error #128

Closed
austinpray opened this issue Jun 27, 2020 · 55 comments
Assignees
Labels
bug Something isn't working

Comments

@austinpray
Copy link

austinpray commented Jun 27, 2020

TL;DR

Using GoogleCloudPlatform/github-actions/setup-gcloud@master with ubuntu-latest. Builds suddenly started failing overnight due to docker-credential-gcloud throwing a version OPENSSL_1_1_1' not found error.

Expected behavior

docker-credentials-gcloud should allow me to pull an image from my gcp repo

Observed behavior

docker-credentials-gcloud fails complaining about version OPENSSL_1_1_1' not found and build fails.

https://github.com/austinpray/kaori/pull/134/checks?check_run_id=814820107#step:6:90

ERROR: gcloud failed to load: /tmp/_MEIhQjalI/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so)

👉 Reverting to version 297.0.1 as in austinpray/kaori#133 makes the build pass again.

Reproduction

Here is me fixing it by pinning the gcloud version to an older one: austinpray/kaori#133

Here is me reproducing the issue by unpinning the version: austinpray/kaori#134 (https://github.com/austinpray/kaori/pull/134/checks?check_run_id=814820107#step:6:90)

Current runner version: '2.263.0'
Operating System
  Ubuntu
  18.04.4
  LTS
Virtual Environment
  Environment: ubuntu-18.04
  Version: 20200621.1
  Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20200621.1/images/linux/Ubuntu1804-README.md

Action YAML

name: CI

on:
  push:
    branches: [ master ]
    paths-ignore:
      - workloads/**
      - fluxcd/**
  pull_request:

jobs:

  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
      with:
        project_id: ${{ secrets.GCLOUD_PROJECT_ID }}
        service_account_key: ${{secrets.GOOGLE_APPLICATION_CREDENTIALS}}
        export_default_credentials: true
    - run: gcloud auth configure-docker
    - name: Create a .env file
      env:
        SLACK_API_TOKEN: ${{secrets.SLACK_API_TOKEN}}
        SLACK_SIGNING_SECRET: ${{secrets.SLACK_SIGNING_SECRET}}
        SLACK_VERIFICATION_TOKEN: ${{secrets.SLACK_VERIFICATION_TOKEN}}
        GCLOUD_SERVICE_ACCOUNT_INFO: ${{secrets.GCLOUD_SERVICE_ACCOUNT_INFO}}
        IMAGES_BUCKET_GCLOUD: ${{secrets.IMAGES_BUCKET_GCLOUD}}
      run: ./scripts/create-dotenv-from.py .env.example > .env
    - name: Build the Docker image
      run: |
        docker pull $CACHE_FROM_IMAGE || true
        make
      env:
        CACHE_FROM_IMAGE: us.gcr.io/${{secrets.GCLOUD_PROJECT_ID}}/kaori/kaori:master
    - name: Run the tests
      run: ./tests/run.sh
    - name: Upload coverage to Codecov
      uses: codecov/codecov-action@v1
      with:
        token: ${{ secrets.CODECOV_TOKEN }}
    - name: Run the card sim
      run: make test-sims
    - name: Push to GCR
      if: github.ref == 'refs/heads/master'
      run: |
        docker tag austinpray/kaori/kaori us.gcr.io/${{secrets.GCLOUD_PROJECT_ID}}/kaori/kaori:master-${{github.sha}}
        docker tag austinpray/kaori/kaori us.gcr.io/${{secrets.GCLOUD_PROJECT_ID}}/kaori/kaori:master
        docker push us.gcr.io/${{secrets.GCLOUD_PROJECT_ID}}/kaori/kaori:master
        docker push us.gcr.io/${{secrets.GCLOUD_PROJECT_ID}}/kaori/kaori:master-${{github.sha}}

Repository

austinpray/kaori#133

If you want temporary access to this repo for debugging and such: I can give it to you. This is just a silly side project.

@aDisplayName
Copy link

Same here. When running docker-compose with latest google SDK, showing the similiar message when pulling base image from gcr.
The workaround by using 297.0.1 got it work.

@bharathkkb
Copy link
Contributor

bharathkkb commented Jul 6, 2020

Hi @austinpray
Looking at the runner env OPENSSL_1_1_1 seems to be installed. Does 299.0.0 work? I will try to see if something on the gcloud side may have changed.
related actions/runner-images#1106

@bharathkkb
Copy link
Contributor

I tried a minimal repro here based on actions/runner-images#1106 which seemed to work with 298.

anayden added a commit to neuro-inc/platform-registry-api that referenced this issue Jul 6, 2020
anayden added a commit to neuro-inc/platform-registry-api that referenced this issue Jul 7, 2020
* Initial Workflows integration

* Add make init

* Remove make init

* Secret migration

* Secret migration 2

* Secret migration 3

* Secret migration 4

* Secret migration 5

* Secret migration 6

* Secret migration 7

* Secret migration 8

* Add Docker client step

* Add Docker client step 2

* Add Docker client step 3

* Add Docker client step 4

* Add Docker client step 5

* Add Docker client step 6

* Add Docker client step 7

* Add Docker client step 8

* Add Docker client step 9

* Add Docker client step 10

* Add Docker client step 11

* Add Docker client step 12

* Add Docker client step 13

* Add Docker client step 14

* Add Docker client step 15

* Add Docker client step 16

* Add Docker client step 17

* Add Docker client step 18

* Add Docker client step 19

* Add Docker client step 20

* Add Docker client step 21

* Add Docker client step 22

* Add Docker client step 23

* Add Docker client step 24

* Add Docker client step 25

* Add Docker client step 26

* Add Docker client step 27

* Add Docker client step 28

* Setup AWS tests 1

* Setup AWS tests 2

* Setup AWS tests 3

* Add AWS and Artifactory 3

* Add AWS and Artifactory 4

* Add AWS and Artifactory 5

* Add AWS and Artifactory 6

* Add AWS and Artifactory 7

* Add AWS and Artifactory 8

* Add AWS and Artifactory 9

* Add AWS and Artifactory 10

* Add AWS and Artifactory 11

* Add AWS and Artifactory 12

* Add AWS and Artifactory 13

* Add AWS and Artifactory final

* Add AWS and Artifactory final

* Add AWS and Artifactory - remove debug output

* Address code review comments

* Put GKE auth back

* Pin gcloud tools version to fix an error

See google-github-actions/setup-gcloud#128

* Disable GKE login once again

* Enable GKE login for good

* Remove test and replace it with aws_test
@YAmikep
Copy link

YAmikep commented Jul 14, 2020

Same issue. 298 did not seem to work for me. 297.0.01 worked.

@robbrit
Copy link

robbrit commented Jul 14, 2020

Confirming that this issue also exists in Ubuntu 20.04 with gcloud version 301.0.0.

To fix:

sudo apt install google-cloud-sdk=297.0.1-0

@austinpray
Copy link
Author

austinpray commented Jul 14, 2020

Confirming that the original sighting of this still happens on the latest runner: https://github.com/austinpray/kaori/runs/869998041?check_suite_focus=true#step:6:84

# github actions runner
Current runner version: '2.267.1'
Run GoogleCloudPlatform/github-actions/setup-gcloud@master
  with:
    project_id: ***
    service_account_key: ***
    export_default_credentials: true
    version: latest

Results in gcloud version 301.0.0

@polaroidkidd
Copy link

Confirming that this issue also exists in Ubuntu 20.04 with gcloud version 301.0.0.

To fix:

sudo apt install google-cloud-sdk=297.0.1-0

This temporary workaround also works on Ubuntu Server 18.04.4.

@rgeraads
Copy link

What worked for us was to use the env var:

CLOUDSDK_PYTHON: /usr/bin/python

@mjhoffman65
Copy link

Ran into this error as well on Ubuntu 18.04.4 LTS

@seilagamo
Copy link

I get this error when, for example, I execute docker-compose build, but if I exectue this command with sudo it works.

@jtrh
Copy link

jtrh commented Jul 23, 2020

I get this error when, for example, I execute docker-compose build

I also got the OPENSSL_1_1_1 error when executing docker-compose build with google-cloud-sdk Debian package version 302.0.0-0. Reverting to version 301.0.0-0 did not fix the problem, but reverting to 300.0.0-0 did.

System information:

  • OS:
    # lsb_release --description
    Description:  Ubuntu 20.04 LTS
    
  • Google Cloud SDK installed through apt-get.
  • Docker Compose:
    # docker-compose version
    docker-compose version 1.26.2, build eefe0d31
    docker-py version: 4.2.2
    CPython version: 3.7.7
    OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
    

@sorliem
Copy link

sorliem commented Jul 23, 2020

@jtrh getting the same error as you while doing docker-compose build. I tried prefixing it with CLOUDSDK_PYTHON=$(which python3) didn't work either, same error.

System info:

$ gcloud -v
Google Cloud SDK 302.0.0
alpha 2020.07.17
beta 2020.07.17
bq 2.0.58
cloud-build-local
core 2020.07.17
gsutil 4.52
kubectl 1.15.11
pubsub-emulator 0.1.0

$ lsb_release --all
No LSB modules are available.
Distributor ID: LinuxMint
Description:    Linux Mint 19.3 Tricia
Release:        19.3
Codename:       tricia

$ docker-compose version
docker-compose version 1.24.1, build 4667896b
docker-py version: 3.7.3
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.1.0j  20 Nov 2018

EDIT: this fixed it for me openssl/openssl#5845 (comment)

@lricoy
Copy link

lricoy commented Jul 27, 2020

I was having this problem using the google/cloud-sdk image on CircleCI.

My workaround was changing the docker-compose install method from the one recommended on the CircleCI docs to:

pip3 install docker-compose --upgrade

@bharathkkb
Copy link
Contributor

Hi @austinpray
Could you add a gcloud info step with a build that errors out. I am unable to repro here with latest versions and 301.0.0. If you have any tips on a reproduction, that would also be great.

@austinpray
Copy link
Author

@bharathkkb sure thing! here is the output

gcloud info output
2020-07-30T14:25:23.0340993Z Google Cloud SDK [303.0.0]
2020-07-30T14:25:23.0341792Z 
2020-07-30T14:25:23.0343161Z Platform: [Linux, x86_64] uname_result(system='Linux', node='fv-az117', release='5.3.0-1032-azure', version='#33~18.04.1-Ubuntu SMP Fri Jun 26 15:01:15 UTC 2020', machine='x86_64', processor='x86_64')
2020-07-30T14:25:23.0343851Z Locale: ('en_US', 'UTF-8')
2020-07-30T14:25:23.0344499Z Python Version: [3.6.9 (default, Jul 17 2020, 12:50:27)  [GCC 8.4.0]]
2020-07-30T14:25:23.0344994Z Python Location: [/usr/bin/python3]
2020-07-30T14:25:23.0345562Z Site Packages: [Disabled]
2020-07-30T14:25:23.0345871Z 
2020-07-30T14:25:23.0346338Z Installation Root: [/opt/hostedtoolcache/gcloud/303.0.0/x64]
2020-07-30T14:25:23.0346770Z Installed Components:
2020-07-30T14:25:23.0347217Z   bq: [2.0.58]
2020-07-30T14:25:23.0347670Z   gsutil: [4.52]
2020-07-30T14:25:23.0348070Z   core: [2020.07.24]
2020-07-30T14:25:23.0348634Z System PATH: [/opt/hostedtoolcache/gcloud/303.0.0/x64/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/usr/share/rust/.cargo/bin:/home/runner/.config/composer/vendor/bin:/home/runner/.dotnet/tools:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games]
2020-07-30T14:25:23.0349672Z Python PATH: [/opt/hostedtoolcache/gcloud/303.0.0/x64/lib/third_party:/opt/hostedtoolcache/gcloud/303.0.0/x64/lib:/usr/lib/python36.zip:/usr/lib/python3.6:/usr/lib/python3.6:/usr/lib/python3.6/lib-dynload]
2020-07-30T14:25:23.0350249Z Cloud SDK on PATH: [True]
2020-07-30T14:25:23.0350685Z Kubectl on PATH: [/usr/bin/kubectl]
2020-07-30T14:25:23.0351473Z 
2020-07-30T14:25:23.0352167Z WARNING: There are other instances of the Google Cloud Platform tools on your system PATH.
2020-07-30T14:25:23.0352927Z   /usr/lib/google-cloud-sdk/bin/git-credential-gcloud.sh
2020-07-30T14:25:23.0353332Z   /usr/lib/google-cloud-sdk/bin/bq
2020-07-30T14:25:23.0353746Z   /usr/lib/google-cloud-sdk/bin/docker-credential-gcloud
2020-07-30T14:25:23.0354120Z   /usr/lib/google-cloud-sdk/bin/anthoscli
2020-07-30T14:25:23.0354499Z   /usr/lib/google-cloud-sdk/bin/gcloud
2020-07-30T14:25:23.0354831Z   /usr/lib/google-cloud-sdk/bin/gsutil
2020-07-30T14:25:23.0354971Z 
2020-07-30T14:25:23.0355182Z Installation Properties: [/opt/hostedtoolcache/gcloud/303.0.0/x64/properties]
2020-07-30T14:25:23.0355416Z User Config Directory: [/home/runner/.config/gcloud]
2020-07-30T14:25:23.0355719Z Active Configuration Name: [default]
2020-07-30T14:25:23.0355946Z Active Configuration Path: [/home/runner/.config/gcloud/configurations/config_default]
2020-07-30T14:25:23.0356116Z 
2020-07-30T14:25:23.0357169Z Account: [github-actions@***.iam.gserviceaccount.com]
2020-07-30T14:25:23.0357605Z Project: [***]
2020-07-30T14:25:23.0357692Z 
2020-07-30T14:25:23.0357961Z Current Properties:
2020-07-30T14:25:23.0358150Z   [core]
2020-07-30T14:25:23.0358540Z     account: [github-actions@***.iam.gserviceaccount.com]
2020-07-30T14:25:23.0358811Z     disable_usage_reporting: [True]
2020-07-30T14:25:23.0359161Z     project: [***]
2020-07-30T14:25:23.0359352Z   [metrics]
2020-07-30T14:25:23.0359767Z     environment: [github-actions-setup-gcloud]
2020-07-30T14:25:23.0359866Z 
2020-07-30T14:25:23.0360107Z Logs Directory: [/home/runner/.config/gcloud/logs]
2020-07-30T14:25:23.0360335Z Last Log File: [/home/runner/.config/gcloud/logs/2020.07.30/14.25.16.618342.log]
2020-07-30T14:25:23.0360494Z 
2020-07-30T14:25:23.0360800Z git: [b'git version 2.27.0']
2020-07-30T14:25:23.0361171Z ssh: [b'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017']
2020-07-30T14:25:23.0361335Z 
2020-07-30T14:25:23.0361688Z 

@bharathkkb I did notice that you were using docker pull instead of docker-compose pull

@bharathkkb
Copy link
Contributor

Thanks @austinpray! I just saw the docker-compose commands in your run.sh :)

@rodrigobb
Copy link

Thank you @sorliem. The solution you linked also worked for me (gcloud version 304).

Also remark that adding

LD_LIBRARY_PATH=/usr/local/lib;

in the command line didn't worked. Variable has to be exported.

@bharathkkb
Copy link
Contributor

I was able to reproduce this here. I was also able to fix it with the temporary workaround from openssl/openssl#5845 (comment) suggest by @sorliem.

I will continue to look into a possible long term fix.

@eino-makitalo
Copy link

Thank you @sorliem. The solution you linked also worked for me (gcloud version 304).

Also remark that adding

LD_LIBRARY_PATH=/usr/local/lib;

in the command line didn't worked. Variable has to be exported.
I have this very same problem after updating my Ubuntu 18.04 environment gcloud but
when setting D_LIBRARY_PATH=/usr/local/lib I can run my docker-compose build commands OK. Thx!

@julianrutten
Copy link

I do not know if this is helpful if not excuse me: the same just happened to me after upgrading to 305.0.0. I am not sure what version I was running before but it was certainly 30x. Running Ubuntu 20.04 LTS. Exact error message below:

ERROR: gcloud failed to load: /tmp/_MEI3WIF2B/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so)

@jacksonwbrito
Copy link

I got the same error on Ubuntu 18.04 and gcloud 304.0.0

export CLOUDSDK_PYTHON=/usr/bin/python worked for me as well.

@mikzuit
Copy link

mikzuit commented Aug 25, 2020

@jtrh getting the same error as you while doing docker-compose build. I tried prefixing it with CLOUDSDK_PYTHON=$(which python3) didn't work either, same error.

System info:

$ gcloud -v
Google Cloud SDK 302.0.0
alpha 2020.07.17
beta 2020.07.17
bq 2.0.58
cloud-build-local
core 2020.07.17
gsutil 4.52
kubectl 1.15.11
pubsub-emulator 0.1.0

$ lsb_release --all
No LSB modules are available.
Distributor ID: LinuxMint
Description:    Linux Mint 19.3 Tricia
Release:        19.3
Codename:       tricia

$ docker-compose version
docker-compose version 1.24.1, build 4667896b
docker-py version: 3.7.3
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.1.0j  20 Nov 2018

EDIT: this fixed it for me openssl/openssl#5845 (comment)

Thanks @sorliem
Confirmed Workaround fix mine with:

gcloud -v
Google Cloud SDK 307.0.0
bq 2.0.59
core 2020.08.21
gsutil 4.53

lsb_release --all
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal

docker-compose version
docker-compose version 1.26.2, build eefe0d31
docker-py version: 4.2.2
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019

python --version
Python 3.8.2

@nikolaik
Copy link

nikolaik commented Aug 26, 2020

Why is LD_LIBRARY_PATH set to /tmp/_MEI3WIF2B/? Is this a bug with vendored python dependencies in docker-compose?

@xakraz
Copy link

xakraz commented Aug 26, 2020

I can confirm that @bharathkkb workaround mentioned with the export LD_LIBRARY_PATH=/usr/local/lib works for me !

By the way, to make things a bit harder to debug,I am using https://github.com/asdf-vm/asdf to install my stuff.
But the workaround works

$ asdf list

argocd
  1.5.4
docker-compose
  1.26.0
gcloud
  293.0.0
gohugo
  extended_0.72.0
helm
  3.2.0
kubectl
  1.15.11
  1.16.11
kubesec
  0.9.2
kustomize
  3.5.5
  3.8.1
sops
  v3.5.0
stern
  1.11.0
terraform
  0.12.16
  0.12.18

So with

docker-compose
  1.26.0
gcloud
  293.0.0

@henryse
Copy link

henryse commented Nov 5, 2020

Seeing this in Linux odin 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 GNU/Linux

Forced to revert back to python2.

@ThomWright
Copy link

ThomWright commented Nov 24, 2020

I've started getting this after updating docker-compose on my machine. It happens if I run docker-compose pull.

I was previously on a much older version of docker-compose, which worked. I can't remember which version, but it might have been installed as long ago as 2017.

EDIT: LD_LIBRARY_PATH=/usr/lib docker-compose pull my_service works. I suppose I shall use that as a workaround for now. Why gcloud expects to find libssl in /tmp/... is... not something I understand.

Error:

ERROR: gcloud failed to load: /tmp/_MEIMOBjio/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so)
    gcloud_main = _import_gcloud_main()
    import googlecloudsdk.gcloud_main
    from googlecloudsdk.api_lib.iamcredentials import util as iamcred_util
    from googlecloudsdk.api_lib.util import apis_internal
    from googlecloudsdk.core import properties
    from googlecloudsdk.core import config
    from oauth2client import client
    from oauth2client import transport
    import httplib2
    from httplib2.python3.httplib2 import *
    import ssl
    import _ssl             # if we can't import it, let the error propagate

This usually indicates corruption in your gcloud installation or problems with your Python interpreter.

Please verify that the following is the path to a working Python 2.7 or 3.5+ executable:
    /usr/bin/python3

The above is truncated. Full error here.

Relevant versions:

/usr/bin/python3 --version
Python 3.6.9

I assume the Python executable being used is the one in /usr/bin/python3, since that's what's in the error message. However, for completeness:

which python3
/home/thom/.pyenv/shims/python3

python3 --version
Python 3.6.9

gcloud version
Google Cloud SDK 319.0.0
alpha 2020.11.13
beta 2020.11.13
bq 2.0.62
core 2020.11.13
gsutil 4.55
kubectl 1.16.13

lsb_release -d
Description: Ubuntu 18.04.5 LTS

docker-compose version
docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019

openssl version
OpenSSL 1.1.1 11 Sep 2018

which openssl
/usr/bin/openssl

locate libssl.so.1.1
/snap/core18/1885/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/snap/core18/1932/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/snap/gnome-3-28-1804/128/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/snap/gnome-3-28-1804/145/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/snap/gnome-3-34-1804/36/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/snap/gnome-3-34-1804/60/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/usr/lib/x86_64-linux-gnu/libssl.so.1.1

@davidmoshal
Copy link

The magical incantation

export LD_LIBRARY_PATH=/usr/local/lib

as described above, worked for me.

@vicpara
Copy link

vicpara commented Dec 3, 2020

Same problem here. Is there any known fix for the gihub action as described by the OP ?

@martin-dmtrv
Copy link

export LD_LIBRARY_PATH=/usr/local/lib worked for me too.

I had the exact same problem. The above mentioned export worked for me.

Ubuntu 20.04
Google Cloud SDK 322.0.0

Thanks

@dgpc
Copy link

dgpc commented Jan 21, 2021

For docker-compose on GitHub Actions, I'm also working around this by using:

    - uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
      with:
        version: '297.0.1' # https://github.com/google-github-actions/setup-gcloud/issues/128
        ...

It would be great to not be stuck on 297.0.1 forever, the latest version at the time of writing is 324.0.0, any chances of an upstream fix?

@jacek-jablonski
Copy link
Contributor

jacek-jablonski commented Feb 5, 2021

The magical incantation

export LD_LIBRARY_PATH=/usr/local/lib

as described above, worked for me.

With newest version of gcloud (326.0.0) this no longer works :(

@vicpara
Copy link

vicpara commented Feb 6, 2021

Same problem here.
GoogleCloudPlatform/github-actions/setup-gcloud may have been deprecated but it actually works reliably.

@bharathkkb
Copy link
Contributor

Hi @vicpara
There should not be any difference between GoogleCloudPlatform/github-actions/setup-gcloud and google-github-actions/setup-gcloud w.r.t this issue as it is related to gcloud + docker-compose.

@jacek-jablonski I ran a quick test with 327 which seemed to work with the workaround https://github.com/bharathkkb/test-setup-gcloud-128/runs/1876995207 Could you share how it failed in your case?

@vicpara
Copy link

vicpara commented Feb 23, 2021

@bharathkkb The issue I have with this new version is that it breaks gcloud auth configure-docker.

Here is my use case below.

build-deploy:
    needs: bump-version
    name: "Build and Deploy"
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Setup GCP credentials
        uses: google-github-actions/setup-gcloud@master
        with:
          project_id: ${{ secrets.GCLOUD_PROJECT_ID }}
          service_account_key: ${{ secrets.GCP_SA_KEY_JSON }}
          export_default_credentials: true

      - name: Docker pull
        run: |
          gcloud auth configure-docker
          docker-compose pull

The error I get is:

Run gcloud auth configure-docker
  gcloud auth configure-docker
  docker-compose pull
  shell: /bin/bash -e ***0***
  env:
    GKE_PROJECT_ID: ***
    GKE_EMAIL: ***
    GITHUB_SHA: 819aa0a83ff3641e84c2ce6676260186c4344330
    GITHUB_BRANCH: refs/heads/master
    GKE_ZONE: ***
    GKE_CLUSTER: ***
    REGISTRY_HOSTNAME: eu.gcr.io
    DEPLOYMENT_ENV: rc
    DEPLOYMENT_COMMIT_SHA: 819aa0a83ff3641e84c2ce6676260186c4344330
    SEM_VERSION: ***
    VERSION_TIMESTAMP: 1614099296
    BUILDHASH: 819aa0a83ff3641e8412de6676260186c4344330
    BUILDTAG: 0.16.75.0
    CLOUDSDK_METRICS_ENVIRONMENT: github-actions-setup-gcloud
    GCLOUD_PROJECT: ***
    GOOGLE_APPLICATION_CREDENTIALS: /home/runner/work/core/core/c7331fe3-c987-4291-9289-4b0ac681b435
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
After update, the following will be written to your Docker config file
 located at [/home/runner/.docker/config.json]:
 ***
  "credHelpers": ***
    "gcr.io": "gcloud",
    "us.gcr.io": "gcloud",
    "eu.gcr.io": "gcloud",
    "asia.gcr.io": "gcloud",
    "staging-k8s.gcr.io": "gcloud",
    "marketplace.gcr.io": "gcloud"
  ***
***

Do you want to continue (Y/n)?  
Docker configuration file updated.
Pulling xxxxxxxxxx        ... 
Pulling xxxxxxxxxx    ... 
Pulling xxxxxxxxxx ... 
Pulling xxxxxxxxxx ... 
Pulling xxxxxxxxxx-service        ... 
Pulling xxxxxxxxxx-service        ... 
ERROR: gcloud failed to load: /tmp/_MEINIeVty/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so)
    gcloud_main = _import_gcloud_main()
    import googlecloudsdk.gcloud_main
    from googlecloudsdk.calliope import base
    from googlecloudsdk.calliope import arg_parsers
    from googlecloudsdk.core import log
ERROR: for artifacts-service  Credentials store error: StoreError('Credentials store docker-credential-gcloud exited with "".')
Traceback (most recent call last):
  File "docker/credentials/store.py", line 80, in _execute
  File "subprocess.py", line 411, in check_output
  File "subprocess.py", line 512, in run
subprocess.CalledProcessError: Command '['/opt/hostedtoolcache/gcloud/329.0.0/x64/bin/docker-credential-gcloud', 'get']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docker/auth.py", line 264, in _resolve_authconfig_credstore
  File "docker/credentials/store.py", line 35, in get
  File "docker/credentials/store.py", line 93, in _execute
docker.credentials.errors.StoreError: Credentials store docker-credential-gcloud exited with "".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 80, in main
  File "compose/cli/main.py", line 192, in perform_command
  File "compose/metrics/decorator.py", line 18, in wrapper
  File "compose/cli/main.py", line 828, in pull
  File "compose/project.py", line 763, in pull
  File "compose/project.py", line 813, in parallel_pull
  File "compose/parallel.py", line 106, in parallel_execute
  File "compose/parallel.py", line 204, in producer
  File "compose/project.py", line 795, in pull_service
  File "compose/service.py", line 1236, in _do_pull
  File "docker/api/image.py", line 411, in pull
  File "docker/auth.py", line 48, in get_config_header
  File "docker/auth.py", line 324, in resolve_authconfig
  File "docker/auth.py", line 235, in resolve_authconfig
  File "docker/auth.py", line 281, in _resolve_authconfig_credstore
docker.errors.DockerException: Credentials store error: StoreError('Credentials store docker-credential-gcloud exited with "".')
Error: Process completed with exit code 255.

@guywilsonjr
Copy link

The magical incantation

export LD_LIBRARY_PATH=/usr/local/lib

as described above, worked for me.

With newest version of gcloud (326.0.0) this no longer works :(

Linux Mint 20.10 This is the cure in combination with
export CLOUDSDK_PYTHON=/usr/bin/python3

@hidai
Copy link

hidai commented May 18, 2021

297.0.1 for arm64 doesn't seem to be available via apt-get anymore...

340.0.0-0 works with

export LD_LIBRARY_PATH=/usr/local/lib

@JoannaFalkowska
Copy link

JoannaFalkowska commented Jun 2, 2021

None of the solutions mentioned above helped me.

Before finally finding my own workaround, I tried:

  • downgrading gcloud to v297
  • upgrading gcloud to v340 (it was 330 initially)
  • export CLOUDSDK_PYTHON=/usr/bin/python3 and export LD_LIBRARY_PATH=/usr/local/lib (both were already in my .zshrc)
  • pip3 install docker-compose --upgrade
  • changed my local version of Python using pyenv to 3.7.10 which is the same as CPython used by my docker-compose (inspired by this answer)

For anyone using WSL, and probably future me:

What eventually helped me was realizing that because my Ubuntu runs docker-compose through WSL and the entire Docker runs on Windows, I might have some luck setting up the same container in Windows, and then just getting it back up from my usual WSL console and Ubuntu filesystem.

I cloned my repo to Windows filesystem, installed gcloud for Windows, set up gcloud authentication, ran docker-compose up from cmd, and sure enough Docker pulled the images and the container built just fine. Now that my Docker for Windows has all the dependency images, I don't have any trouble running the container from inside WSL.

@SimoneLazzaris
Copy link

Debian 10 here, with gcloud v346 and python 3.7.3 (from debian packages).
the export workaround didn't worked: gcloud still tries to find libraries in /tmp/something

@dthanhtuan
Copy link

dthanhtuan commented Jul 10, 2021

#128 (comment)
CLOUDSDK_PYTHON=/usr/bin/python worked for me while export CLOUDSDK_PYTHON=$(which python3) didn't work.

@LiamAttClarke
Copy link

I ran into this issue when running docker-compose in a GCP VM instance running Debian 10. Downgrading to Debian 9 resolved the issue.

@dgpc
Copy link

dgpc commented Aug 7, 2021

FYI Today I removed the v297.0.1 workaround mentioned here, and things are once again working on the latest version of ubuntu & glcoud; I also moved to the new google-github-actions/setup-gcloud@master repository instead of uses: GoogleCloudPlatform/github-actions/setup-gcloud@master

@zeppelinen
Copy link

Just happened with github actions ubuntu:latest and command gcloud auth configure-docker -q

@aridjemana
Copy link

Everyone here put their lucky number with this sdk. i will start from the most like answer

@mattwelke
Copy link

image

@Kartones
Copy link

Kartones commented Oct 4, 2021

Thanks for the solutions posted in the comments.

In my case, what worked (gcloud sdk version from ~1st of October 2021) with Ubuntu 20.04 that had Python 3.8 installed, was:

  1. installing app-engine-python (if I'm correct, installs Python 2.7)
  2. export CLOUDSDK_PYTHON=/usr/bin/python2.7

Tried with to python 3 path but was still giving me the OPENSSL_1_1_1 not found error, but pinning/forcing to Python 2 is so far working without issues.

@LegolasVzla
Copy link

LegolasVzla commented Nov 3, 2021

gcloud --version

Google Cloud SDK 362.0.0
alpha 2021.10.21
beta 2021.10.21
bq 2.0.71
core 2021.10.21
gsutil 5.4

I tried with export CLOUDSDK_PYTHON=/usr/bin/python3 export LD_LIBRARY_PATH=/usr/local/lib but neither of them worked for me. I'm facing this error when i do sudo docker-compose pull my_image, but if I try withoud sudo, i got:

ERROR: for web  unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

ERROR: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

I fixed with authentication in gcloud withoud sudo:

gcloud auth login
gcloud config set project my_project_id
gcloud auth configure-docker

Then, I could do: docker-compose pull my_image

@lorainegarutti
Copy link

lorainegarutti commented Nov 11, 2021

It also happened to me with Linux (Ubuntu 20.04) and it was related with sudo configuration.

I solved with sudo docker-compose xxxx

Even if I make a groupadd configuration, it continues showing this error 😭

@lcampos
Copy link

lcampos commented Jan 31, 2022

Just ran into this today and the previously suggested fixes didn't work for me. I ended up re-installing docker-compose which sorted out this issue (pip install docker-compose)

Config:

GH Action runner 2.287.1
Ubuntu v20.04.3
google-github-actions/setup-gcloud@v0.3.0
Google Cloud SDK 370.0.0
Python 3.8.10

@sethvargo
Copy link
Member

Hi folks - given Python 2 is deprecated, I'm going to close out this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Development

No branches or pull requests