Skip to content

Commit

Permalink
Upgrade to go version 1.20.5 (#1196)
Browse files Browse the repository at this point in the history
Brief list of changes

* Changes gcsfuse binary to be statically-linked, removes dependency
   on libc.
* Update tools/package_gcsfuse_docker/Dockerfile base to golang:1.20.5
* Switch from os.user.Current() to os.Getuid() and os.Getgid()
* Add CGO_ENABLED=0 to disable CGO to tools/build_gcsfuse

Other related changes.
  • Loading branch information
gargnitingoogle committed Jul 20, 2023
1 parent 640bed7 commit 908c1f1
Show file tree
Hide file tree
Showing 13 changed files with 22 additions and 35 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,11 @@ jobs:
run: sudo apt-get update && sudo apt-get install -y fuse3 libfuse-dev
- name: Build
run: |
go build ./...
CGO_ENABLED=0 go build ./...
go install ./tools/build_gcsfuse
build_gcsfuse . /tmp ${GITHUB_SHA}
- name: Test
run: go test -p 1 -count 1 -v -cover ./...
run: CGO_ENABLED=0 go test -p 1 -count 1 -v -cover ./...
lint:
name: Lint
runs-on: ubuntu-20.04
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
# Mount the gcsfuse to /mnt/gcs:
# > docker run --privileged --device /fuse -v /mnt/gcs:/gcs:rw,rshared gcsfuse

FROM golang:1.20.4-alpine as builder
FROM golang:1.20.5-alpine as builder

RUN apk add git

Expand Down
1 change: 0 additions & 1 deletion docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ This page enumerates some common user facing issues around GCSFuse and also disc
| Generic Mounting Issue | Most of the common mount point issues are around permissions on both local mount point and the Cloud Storage bucket. It is highly recommended to retry with --foreground --debug_fuse --debug_fs --debug_gcs --debug_http flags which would provide much more detailed logs to understand the errors better and possibly provide a solution. |
| Mount successful but files not visible | Try mounting the gcsfuse with --implicit-dir flag. Read the [semantics](https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/semantics.md) to know the reasoning. |
| Mount failed with fusermount3 exit status 1 | It comes when the bucket is already mounted in a folder and we try to mount it again. You need to unmount first and then remount. |
| Mount failed with error: Current requires cgo or $USER set in environment | It comes when we try mounting by building the gcsfuse codebase. To fix this, build the gcsfuse package by enabling the CGO_ENABLED flag in the go env and then mount back.<ol type="a"><li> Check the current value using - ```go env``` command. </li><li>If it is unset, set this using - ```export CGO_ENABLED=1``` command. </li></ol> |
| Mount get stuck with error: DefaultTokenSource: google: could not find default credentials | Run ```gcloud auth application-default login``` command to fetch default credentials to the VM. This will fetch the credentials to the following locations: <ol type="a"><li>For linux - $HOME/.config/gcloud/application_default_credentials.json</li><li>For windows - %APPDATA%/gcloud/applicateion_default_credentials.json </li></ol> |
| Input/Output Error | It’s a generic error, but the most probable culprit is the bucket not having the right permission for Cloud Storage FUSE to operate on. Ref - [here](https://stackoverflow.com/questions/36382704/gcsfuse-input-output-error) |
| Generic NO_PUBKEY Error - while installing Cloud Storage FUSE on ubuntu 22.04 | It happens while running - ```sudo apt-get update``` - working on installing Cloud Storage FUSE. You just have to add the pubkey you get in the error using the below command: ```sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys <PUBKEY> ``` And then try running ```sudo apt-get update``` |
Expand Down
28 changes: 7 additions & 21 deletions internal/perms/perms.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,35 +17,21 @@ package perms

import (
"fmt"
"os/user"
"strconv"
"os"
)

// MyUserAndGroup returns the UID and GID of this process.
func MyUserAndGroup() (uid uint32, gid uint32, err error) {
// Ask for the current user.
user, err := user.Current()
if err != nil {
err = fmt.Errorf("Fetching current user: %w", err)
return
}

// Parse UID.
uid64, err := strconv.ParseUint(user.Uid, 10, 32)
if err != nil {
err = fmt.Errorf("Parsing UID (%s): %w", user.Uid, err)
return
}
signed_uid := os.Getuid()
signed_gid := os.Getgid()

// Parse GID.
gid64, err := strconv.ParseUint(user.Gid, 10, 32)
if err != nil {
err = fmt.Errorf("Parsing GID (%s): %w", user.Gid, err)
if signed_gid == -1 || signed_uid == -1 {
err = fmt.Errorf("Failed to get uid/gid. UID = %d, GID = %d", signed_uid, signed_gid)
return
}

uid = uint32(uid64)
gid = uint32(gid64)
uid = uint32(signed_uid)
gid = uint32(signed_gid)

return
}
2 changes: 1 addition & 1 deletion perfmetrics/scripts/compare_fuse_types_using_fio.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ def _install_gcsfuse_source(gcs_bucket, gcsfuse_flags) -> None:
os.system(f'''git clone {GCSFUSE_REPO}
mkdir gcs
cd gcsfuse
go run . {gcsfuse_flags} {gcs_bucket} ../gcs
CGO_ENABLED=0 go run . {gcsfuse_flags} {gcs_bucket} ../gcs
cd ..
''')

Expand Down
4 changes: 2 additions & 2 deletions perfmetrics/scripts/continuous_test/gcp_ubuntu/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ echo "Installing git"
sudo apt-get install git
echo "Installing pip"
sudo apt-get install pip -y
echo "Installing go-lang 1.20.4"
wget -O go_tar.tar.gz https://go.dev/dl/go1.20.4.linux-amd64.tar.gz
echo "Installing go-lang 1.20.5"
wget -O go_tar.tar.gz https://go.dev/dl/go1.20.5.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && tar -xzf go_tar.tar.gz && sudo mv go /usr/local
export PATH=$PATH:/usr/local/go/bin
echo "Installing fio"
Expand Down
3 changes: 2 additions & 1 deletion perfmetrics/scripts/ml_tests/pytorch/dino/setup_container.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/bin/bash

wget -O go_tar.tar.gz https://go.dev/dl/go1.20.4.linux-amd64.tar.gz
# Install golang
wget -O go_tar.tar.gz https://go.dev/dl/go1.20.5.linux-amd64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go_tar.tar.gz
export PATH=$PATH:/usr/local/go/bin

Expand Down
2 changes: 1 addition & 1 deletion perfmetrics/scripts/ml_tests/setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# >> source setup.sh

# Go version to be installed.
GO_VERSION=go1.20.4.linux-amd64.tar.gz
GO_VERSION=go1.20.5.linux-amd64.tar.gz

# This function will install the given module/dependency if it's not alredy
# installed.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# and epochs functionality, and runs the model

# Install go lang
wget -O go_tar.tar.gz https://go.dev/dl/go1.20.4.linux-amd64.tar.gz
wget -O go_tar.tar.gz https://go.dev/dl/go1.20.5.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && tar -xzf go_tar.tar.gz && sudo mv go /usr/local
export PATH=$PATH:/usr/local/go/bin

Expand Down
4 changes: 2 additions & 2 deletions perfmetrics/scripts/presubmit_test/pr_perf_test/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ pip install google-cloud
pip install google-cloud-vision
pip install google-api-python-client
pip install prettytable
echo Installing go-lang 1.20.4
wget -O go_tar.tar.gz https://go.dev/dl/go1.20.4.linux-amd64.tar.gz
echo Installing go-lang 1.20.5
wget -O go_tar.tar.gz https://go.dev/dl/go1.20.5.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && tar -xzf go_tar.tar.gz && sudo mv go /usr/local
export PATH=$PATH:/usr/local/go/bin
echo Installing fio
Expand Down
1 change: 1 addition & 0 deletions tools/build_gcsfuse/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ func buildBinaries(dstDir, srcDir, version string, buildArgs []string) (err erro
fmt.Sprintf("GOROOT=%s", runtime.GOROOT()),
fmt.Sprintf("GOPATH=%s", gopath),
fmt.Sprintf("GOCACHE=%s", gocache),
fmt.Sprintf("CGO_ENABLED=0"),

Check failure on line 161 in tools/build_gcsfuse/main.go

View workflow job for this annotation

GitHub Actions / Lint

S1039: unnecessary use of fmt.Sprintf (gosimple)
}

// Build.
Expand Down
2 changes: 1 addition & 1 deletion tools/containerize_gcsfuse_docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ ARG OS_VERSION
ARG OS_NAME

# Image with gcsfuse installed and its package (.deb)
FROM golang:1.20.4 as gcsfuse-package
FROM golang:1.20.5 as gcsfuse-package

RUN apt-get update -qq && apt-get install -y ruby ruby-dev rubygems build-essential rpm fuse && gem install --no-document bundler

Expand Down
2 changes: 1 addition & 1 deletion tools/package_gcsfuse_docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
# Copy the gcsfuse packages to the host:
# > docker run -it -v /tmp:/output gcsfuse-release cp -r /packages /output

FROM golang:1.20.4 as builder
FROM golang:1.20.5 as builder

RUN apt-get update -qq && apt-get install -y ruby ruby-dev rubygems build-essential rpm && gem install --no-document bundler

Expand Down

0 comments on commit 908c1f1

Please sign in to comment.