refactor file I/O & replace io.Copy to vald original faster Copy func…

[kpango]

Signed-off-by: kpango kpango@vdaas.org

Description:

This PR includes 3 changes

1. remove std io.Copy and replace it to internal/io.Copy which uses auto scale pool buffer

❯ go test -bench . -run=NONE -count=1 --benchmem
goos: linux
goarch: amd64
pkg: github.com/vdaas/vald/internal/io
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkStandardIOCopy-16                        411262              3579 ns/op           32816 B/op          3 allocs/op
BenchmarkStandardIOCopyBuffer-16                  366306              3564 ns/op           32816 B/op          3 allocs/op
BenchmarkValdIOCopy-16                           3392606               360.3 ns/op            48 B/op          2 allocs/op
BenchmarkStandardIOCopyParallel-16                439708              2668 ns/op           32816 B/op          3 allocs/op
BenchmarkStandardIOCopyBufferParallel-16          494727              2345 ns/op           32816 B/op          3 allocs/op
BenchmarkValdIOCopyParallel-16                  19977921                65.29 ns/op           50 B/op          2 allocs/op
PASS
ok      github.com/vdaas/vald/internal/io       9.137s

2. use *os.File.Sync() for stability
3. grpc Pool error messages refactor

Related Issue:

How Has This Been Tested?:

Environment:

• Go Version: 1.16.4
• Docker Version: 19.03.8
• Kubernetes Version: 1.18.2
• NGT Version: 1.12.3

Types of changes:

• Bug fix [type/bug]
• New feature [type/feature]
• Add tests [type/test]
• Security related changes [type/security]
• Add documents [type/documentation]
• Refactoring [type/refactoring]
• Update dependencies [type/dependency]
• Update benchmarks and performances [type/bench]
• Update CI [type/ci]

Changes to Core Features:

• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your core changes, as applicable?
• Have you successfully ran tests with your changes locally?

Checklist:

• I have read the CONTRIBUTING document.
• I have checked open Pull Requests for the similar feature or fixes?
• I have added tests and benchmarks to cover my changes.
• I have ensured all new and existing tests passed.
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly.

[vdaas-ci]

[WARNING:INTCFG] Changes in interal/config may require you to change Helm charts. Please check.

[vdaas-ci]

[CHATOPS:HELP] ChatOps commands.

• 🙆‍♀️ /approve - approve
• 💌 /changelog - replace the PR body by changelog details
• 🍱 /format - format codes and add licenses
• ✅ /gen-test - generate test codes
• 🏷️ /label - add labels
• ⏪ /rebase - rebase master
• 🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

[codecov]

Codecov Report

Merging #1261 (ec18553) into master (ffcf4ab) will decrease coverage by 0.48%.
The diff coverage is 22.91%.

@@            Coverage Diff             @@
##           master    #1261      +/-   ##
==========================================
- Coverage   17.96%   17.48%   -0.49%     
==========================================
  Files         501      500       -1     
  Lines       31494    31256     -238     
==========================================
- Hits         5658     5464     -194     
+ Misses      25537    25503      -34     
+ Partials      299      289      -10     

Impacted Files	Coverage Δ
internal/compress/gzip.go	100.00% <ø> (ø)
internal/compress/lz4.go	100.00% <ø> (ø)
internal/compress/zstd.go	100.00% <ø> (ø)
internal/core/converter/tensorflow/tensorflow.go	89.06% <ø> (ø)
internal/db/storage/blob/s3/reader/io/io.go	0.00% <0.00%> (ø)
internal/db/storage/blob/s3/writer/writer.go	100.00% <ø> (ø)
internal/encoding/json/json.go	60.00% <ø> (ø)
internal/errors/config.go	0.00% <0.00%> (ø)
internal/io/io.go	0.00% <ø> (ø)
internal/io/ioutil/ioutil.go	0.00% <0.00%> (ø)
... and 23 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ffcf4ab...ec18553. Read the comment docs.