Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use internode data for DisksInfo, VolsInfo in message pack #10821

Merged
merged 1 commit into from Nov 4, 2020

Conversation

harshavardhana
Copy link
Member

@harshavardhana harshavardhana commented Nov 3, 2020

Description

fix: use internode data for DisksInfo, VolsInfo in message pack

Motivation and Context

Similar to #10775 for fewer memory allocations, since we use
getOnlineDisks() extensively for listing we should optimize it
further.

Additionally, remove all unused walkers from the storage layer

How to test this PR?

Nothing special, all functionality should work the same.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation needed
  • Unit tests needed

Copy link
Contributor

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some unneeded calls, no biggies.

cmd/storage-rest-server.go Show resolved Hide resolved
cmd/storage-rest-server.go Show resolved Hide resolved
Similar to minio#10775 for fewer memory allocations, since we use
getOnlineDisks() extensively for listing we should optimize it
further.

Additionally, remove all unused walkers from the storage layer
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-gateway-azure.sh more...

10821-ef28b85/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:30535
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 86651e0dc7bf:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 8 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 1 minutes and 6 seconds
(4/15) Running aws-sdk-ruby tests ... done in 19 seconds
(5/15) Running awscli tests ... done in 2 minutes and 55 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 3 minutes and 29 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 35 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 55 seconds
{
  "args": {
    "bucketName": "minio-go-test-th36ianstaqak1b9",
    "objectName": "test-object",
    "opts": "",
    "size": -1
  },
  "duration": 1168,
  "function": "PutObject(bucketName, objectName, reader,size,opts)",
  "message": "Unexpected size",
  "name": "minio-go: testPutObjectStreaming",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 41 seconds
{
  "name": "minio-java",
  "function": "uploadObject()",
  "args": "[multi-part upload]",
  "duration": 809,
  "status": "FAIL",
  "error": "java.net.SocketException: Connection reset >>> [java.base/java.net.SocketInputStream.read(SocketInputStream.java:186), java.base/java.net.SocketInputStream.read(SocketInputStream.java:140), okio.Okio$2.read(Okio.java:140), okio.AsyncTimeout$2.read(AsyncTimeout.java:237), okio.RealBufferedSource.indexOf(RealBufferedSource.java:358), okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230), okhttp3.internal.http1.Http1ExchangeCodec.readHeaderLine(Http1ExchangeCodec.java:242), okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.java:213), okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.java:115), okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:94), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142), okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117), okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117), okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142), okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117), okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229), okhttp3.RealCall.execute(RealCall.java:81), io.minio.MinioClient.execute(MinioClient.java:1014), io.minio.MinioClient.uploadPart(MinioClient.java:7781), io.minio.MinioClient.putObject(MinioClient.java:4706), io.minio.MinioClient.uploadObject(MinioClient.java:4931), FunctionalTest.testUploadObject(FunctionalTest.java:706), FunctionalTest.uploadObject_test(FunctionalTest.java:730), FunctionalTest.runTests(FunctionalTest.java:3793), FunctionalTest.main(FunctionalTest.java:4058)]"
}
(9/15) Running minio-js tests ... FAILED in 45 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 79,
  "status": "FAIL",
  "error": "S3Error: The bucket you tried to delete is not empty at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:86:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (_stream_readable.js:682:10) at endReadableNT (_stream_readable.js:1252:12) at processTicksAndRejections (internal/process/task_queues.js:80:21)"
}
(9/15) Running minio-py tests ... FAILED in 19 minutes and 14 seconds
{
  "name": "minio-py:test_thread_safe",
  "status": "FAIL",
  "args": {
    "bucket_name": "minio-py-test-56ae8554-80e2-4c7f-9233-cb3552ee4585",
    "object_name": "47b173c5-2c4e-4dc1-9af4-851825504dab"
  },
  "message": "Sha-sum mismatch on multi-threaded put and get objects",
  "error": "Traceback (most recent call last):\n  File \"/mint/run/core/minio-py/tests.py\", line 145, in _call_test\n    func(log_entry, *args, **kwargs)\n  File \"/mint/run/core/minio-py/tests.py\", line 1666, in test_thread_safe\n    raise exceptions[0]\n  File \"/mint/run/core/minio-py/tests.py\", line 1631, in get_object_and_check\n    'Sha-sum mismatch on multi-threaded put and '\nValueError: Sha-sum mismatch on multi-threaded put and get objects\n",
  "duration": 32285
}
(9/15) Running s3cmd tests ... done in 2 minutes and 21 seconds
(10/15) Running s3select tests ... done in 39 seconds
(11/15) Running security tests ... done in 0 seconds

Executed 11 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

@harshavardhana harshavardhana merged commit 1a1f00f into minio:master Nov 4, 2020
@harshavardhana harshavardhana deleted the fix-linux branch November 4, 2020 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants