lots of socket hang up with S3 #701
Comments
|
@sradu Thanks for filing this issue. We will try to reproduce this issue and get back to you. |
|
@sradu Approximately how many files are in your large directories on AWS? |
|
It was around 150k. Tbh we switched to the raw AWS library and haven't had any problems since. While we haven't seen any errors with the raw version, doing it this way also allowed us to implement a retry mechanism. |
|
The socket hang up usually happens when one hasn't drained the connection is the objectStream promise drained properly? @kannappanr @Colhodm |
|
@kanagarajkm can you take a look at this issue? |
|
Hitting this. Would prefer not to have to move to raw aws library |
|
Im getting this on fPutObject. I don't think that has to do with bucket size. So that taken with the original OP would mean:
ive tried both minio 3.0.0 and 7.0.0 . Works perfectly fine against Minio, but socket hangups against actual S3 |
|
We can see this now in all our mint tests Mint Automation
6380-53a3611/mint-gateway-nas.sh.log:6380-53a3611/mint-xl.sh.log:6380-53a3611/mint-fs.sh.log:6380-53a3611/mint-large-bucket.sh.log:6380-53a3611/mint-dist-xl.sh.log:6380-53a3611/mint-gateway-s3.sh.log:6380-53a3611/mint-gateway-azure.sh.log: |
|
minio/minio#6390 fixes the socket hang issue which is currently happening on the latest master. |
|
@NickStefan i have tested minio-js with AWS S3, i am not seeing any errors. Following is the code var Minio = require("minio");
var minioClient = new Minio.Client({
endPoint: "s3.amazonaws.com",
accessKey: "ACCESS_KEY",
secretKey: "SECRET_KEY"
});
const bucketName = BUCKET_NAME;
const filepath = FILE_PATH;
minioClient.fPutObject(bucketName, "sample3.jpg", filepath, (err, etag) => {
console.log(err, etag);
listObjects();
});
function listObjects() {
const objectsStream = minioClient.listObjectsV2(bucketName, "", false);
objectsStream.on("data", obj => {
if (obj.name) {
console.log(obj.name);
}
});
objectsStream.on("end", () => {
console.log("end");
});
objectsStream.on("error", err => {
console.log(err);
});
}Can you provide the sample code if you are hitting the issue? |
like the other guy i had to move on because this is for work. I ended up patching the minioClient object because i do like being able to use minio locally without s3: The raw AWS function works perfect. |
|
@NickStefan Can you remove minioClient = new Minio.Client({
endPoint: 's3.amazonaws.com',
port: 80,
useSSL: true,
accessKey: S3_KEY,
secretKey: S3_SECRET,
region: S3_BUCKET_REGION,
}); |
|
@NickStefan did you get a chance to test this out? |
|
@sradu can you provide the |
|
@kanagarajkm apologies, I have not yet, but will try and get to it today. |
|
@NickStefan no problem, thank you. |
|
@kaankabalak that did address the socket hangups |
|
@NickStefan Thanks for confirming that it works. Closing the issue based on your feedback. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We're running listObjectsV2 and fGetObject and receiving a lot of "Error: socket hang up" errors. The code is simple:
It happens in large directories quite often, but is sometimes also triggered on smaller ones.
Any suggestions on how to handle this? listObjectsV2 is especially annoying since we can't retry chunks of the operation.
The text was updated successfully, but these errors were encountered: