Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Api and mc return 0 objects / output when downloading from the same dir at the sametime #4842

Closed
rob-killingsworth opened this issue Aug 22, 2017 · 2 comments · Fixed by #4846

Comments

@rob-killingsworth
Copy link

Expected Behavior

I would expect the api and mc to beable to do and ls at the same time as a download from a bucket and if they are unable to access not silently fail i.e. the api returning an empty list and mc to none 0.

Current Behavior

While downloading a 100Mb file on one client if a second does a ls in the same bucket and path it will randomly give no results but does not give an error either

Steps to Reproduce (for bugs)

  1. start to down load a larger file from minio
  2. in a loop run mc ls on the files containing path

Context

I'm trying to use minio in a CI system which needs to check that all the needed files have been uploaded before it can start testing. The files follow a naming convention but i need to beable to use regex when looking for them so need the ls step before trying to download.

Your Environment

Standalone docker minio container running under docker-ce on centos7

@vadmeste was able to quickly reproduce while I was talking to him in the slack channel

@vadmeste
Copy link
Member

Concurrent listing showed this problem for me, like if we run the following code concurrently

package main

import (
	"fmt"

	"github.com/minio/minio-go"
)

func main() {
	s3Client, err := minio.New("localhost:9000", "minio", "minio123", false)
	if err != nil {
		fmt.Println(err)
		return
	}

	// Create a done channel to control 'ListObjects' go routine.
	doneCh := make(chan struct{})

	// Indicate to our routine to exit cleanly upon return.
	defer close(doneCh)

	countObjects := 0

	// List all objects from a bucket-name with a matching prefix.
	for object := range s3Client.ListObjects("testbucket", "", true, doneCh) {
		if object.Err != nil {
			fmt.Println(object.Err)
			return
		}
		countObjects++
	}

	if countObjects == 0 {
		fmt.Printf("WARN: no objects found\n")
	}

	return
}

harshavardhana added a commit to harshavardhana/minio that referenced this issue Aug 22, 2017
Current code was just using io.ReadAll() on an fd()
which might have moved underneath due to a concurrent
read operation. Subsequent read will result in EOF
We should always seek back and read again. pread()
is allowed on all platforms use io.SectionReader to
read from the beginning of the file.

Fixes minio#4842
@harshavardhana harshavardhana self-assigned this Aug 22, 2017
harshavardhana added a commit to harshavardhana/minio that referenced this issue Aug 23, 2017
Current code was just using io.ReadAll() on an fd()
which might have moved underneath due to a concurrent
read operation. Subsequent read will result in EOF
We should always seek back and read again. pread()
is allowed on all platforms use io.SectionReader to
read from the beginning of the file.

Fixes minio#4842
harshavardhana added a commit to harshavardhana/minio that referenced this issue Aug 23, 2017
Current code was just using io.ReadAll() on an fd()
which might have moved underneath due to a concurrent
read operation. Subsequent read will result in EOF
We should always seek back and read again. pread()
is allowed on all platforms use io.SectionReader to
read from the beginning of the file.

Fixes minio#4842
harshavardhana added a commit to harshavardhana/minio that referenced this issue Aug 23, 2017
Current code was just using io.ReadAll() on an fd()
which might have moved underneath due to a concurrent
read operation. Subsequent read will result in EOF
We should always seek back and read again. pread()
is allowed on all platforms use io.SectionReader to
read from the beginning of the file.

Fixes minio#4842
@deekoder deekoder added this to the Edge cache milestone Aug 23, 2017
harshavardhana added a commit to harshavardhana/minio that referenced this issue Aug 23, 2017
Current code was just using io.ReadAll() on an fd()
which might have moved underneath due to a concurrent
read operation. Subsequent read will result in EOF
We should always seek back and read again. pread()
is allowed on all platforms use io.SectionReader to
read from the beginning of the file.

Fixes minio#4842
deekoder pushed a commit that referenced this issue Aug 24, 2017
Current code was just using io.ReadAll() on an fd()
which might have moved underneath due to a concurrent
read operation. Subsequent read will result in EOF
We should always seek back and read again. pread()
is allowed on all platforms use io.SectionReader to
read from the beginning of the file.

Fixes #4842
@lock
Copy link

lock bot commented May 5, 2020

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators May 5, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants