Failed to connect to glusterfs 10.4： connection reset by peer

[hhstu]

Describe the bug
I deployed a standalone Glusterfs with version 10.4 following the guidelines on the official website of Glusterfs. I used CSI to create PVC, but the provider error was:

I0614 02:34:17.599487       1 connection.go:194] GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-8de1d0b5-a323-4271-8cc7-150514ad6940","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
I0614 02:34:17.638991       1 connection.go:200] GRPC response: {}
I0614 02:34:17.639091       1 connection.go:201] GRPC error: rpc error: code = Internal desc = error in fetching volume details Get http://test1:24007/v1/volumes/pvc-8de1d0b5-a323-4271-8cc7-150514ad6940: read tcp 10.0.1.14:34902->10.238.18.110:24007: read: connection reset by peer
I0614 02:34:17.639200       1 controller.go:816] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = error in fetching volume details Get http://test1:24007/v1/volumes/pvc-8de1d0b5-a323-4271-8cc7-150514ad6940: read tcp 10.0.1.14:34902->10.238.18.110:24007: read: connection reset by peer
I0614 02:34:17.639583       1 controller.go:1075] Final error received, removing PVC 8de1d0b5-a323-4271-8cc7-150514ad6940 from claims in progress
W0614 02:34:17.639722       1 controller.go:934] Retrying syncing claim "8de1d0b5-a323-4271-8cc7-150514ad6940", failure 256
E0614 02:34:17.639926       1 controller.go:957] error syncing claim "8de1d0b5-a323-4271-8cc7-150514ad6940": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = error in fetching volume details Get http://test1:24007/v1/volumes/pvc-8de1d0b5-a323-4271-8cc7-150514ad6940: read tcp 10.0.1.14:34902->10.238.18.110:24007: read: connection reset by peer
I0614 02:34:17.640030       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gluster-csi", UID:"8de1d0b5-a323-4271-8cc7-150514ad6940", APIVersion:"v1", ResourceVersion:"997341", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = error in fetching volume details Get http://test1:24007/v1/volumes/pvc-8de1d0b5-a323-4271-8cc7-150514ad6940: read tcp 10.0.1.14:34902->10.238.18.110:24007: read: connection reset by peer

and glusterfs stuatus is ok

[root@glusterfs-test-common.it.ynode.cn ~]# gluster volume info

Volume Name: dv0
Type: Distribute
Volume ID: 026206c7-81c8-4d9a-8738-3264f90d1c78
Status: Started
Snapshot Count: 0
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: test1:/data_brick1/dv0
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
[root@glusterfs-test-common.it.ynode.cn ~]#

Can i get any help, thinks

[shoufuzhang]

I encountered this problem before as well. My solution was to repackage its docker images. The glusterfs version in the official docker images is too low, which causes an error when you try to mount a higher version of gluster.

[shoufuzhang]

cd gluster-csi-driver/pkg/glusterfs
# cat gluster10.repo
[gluster-10]
name=GlusterFS 10 for CentOS 9 Stream
baseurl=https://buildlogs.centos.org/centos/9-stream/storage/x86_64/gluster-10/
enabled=1
gpgcheck=0
# cat Dockerfile
# Copyright 2018 The Gluster CSI Authors.

# Licensed under GNU LESSER GENERAL PUBLIC LICENSE Version 3, 29 June 2007
# You may obtain a copy of the License at
#    https://opensource.org/licenses/lgpl-3.0.html

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#-- Create build environment

FROM golang:1.20 as build

ENV GOPATH="/go/" \
    SRCDIR="/go/src/github.com/gluster/gluster-csi-driver/"
ENV SCRIPTSDIR="${SRCDIR}scripts/"

# Install go tools
ARG GOLANGCILINT_VERSION=latest
COPY scripts/install-go-tools.sh "${SCRIPTSDIR}"
RUN mkdir -p /go/bin; ${SCRIPTSDIR}/install-go-tools.sh

# Vendor dependencies
COPY go.mod go.sum "${SRCDIR}/"
WORKDIR "${SRCDIR}"
RUN go mod download

# Copy source directories
COPY cmd/ "${SRCDIR}/cmd"
COPY pkg/ "${SRCDIR}/pkg"
COPY scripts/ "${SCRIPTSDIR}"

#-- Test phase

ARG RUN_TESTS=1
ARG GO_METALINTER_THREADS=1
ENV TEST_COVERAGE=stdout \
    GO_COVER_DIR=/build/

RUN mkdir /build && \
    CGO_ENABLED=0 GOOS=linux go build -ldflags '-extldflags "-static"' -o /build/glusterfs-csi-driver cmd/glusterfs/main.go
#    [ $RUN_TESTS -eq 0 ] || ( ${SCRIPTSDIR}/lint-go.sh && \
#    ${SCRIPTSDIR}/test-go.sh ) && \
#    CGO_ENABLED=0 GOOS=linux go build -ldflags '-extldflags "-static"' -o /build/glusterfs-csi-driver cmd/glusterfs/main.go

#-- Final container

FROM fedora:37 as final

COPY pkg/glusterfs/gluster10.repo /etc/yum.repos.d/gluster10.repo
# Install dependencies
RUN yum update -y && \
    yum -y install glusterfs-fuse && \
    yum clean all -y && \
    rm -rf /var/cache/yum

# Copy glusterfs-csi-driver from build phase
COPY --from=build /build /

# The version of the driver (git describe --dirty --always --tags | sed 's/-/./2' | sed 's/-/./2')
ARG version="(unknown)"
# Container build time (date -u '+%Y-%m-%dT%H:%M:%S.%NZ')
ARG builddate="(unknown)"

LABEL build-date="${builddate}"
LABEL io.k8s.description="FUSE-based CSI driver for Gluster file access"
LABEL name="glusterfs-csi-driver"
LABEL summary="FUSE-based CSI driver for Gluster file access"
LABEL vcs-type="git"
LABEL vcs-url="https://github.com/gluster/gluster-csi-driver"
LABEL vendor="gluster.org"
LABEL version="${version}"

ENTRYPOINT ["/glusterfs-csi-driver"]