[WIP][FEATURE] AutoRange parsing & format #1664

vdaviot · 2019-02-08T16:18:15Z

This pull-request is related to:

engine: For the docker collector patch, contains more information about why and how
swarmkit: For the API patch

This pull-request contains the necessary changes to parse the autorange key from .yml and .dab files (Contains new v3.8 json schema). It also contains a new format to visualize the predicted values:
$ docker container stats --format autorange

I'm open to any suggestion regarding the implementation.

Cute animal

GordonTheTurtle · 2019-02-08T16:18:17Z

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "master" git@github.com:vdaviot/cli.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842354128976
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

thaJeztah · 2019-02-08T22:49:50Z

cli/command/bundlefile/bundlefile.go

+	WorkingDir *string           			`json:",omitempty"`
+	User       *string           			`json:",omitempty"`
+	Networks   []string          			`json:",omitempty"`
+	AutoRange  map[string]map[string]string `json:",omitempty"`


Haven't looked at the feature as a whole, but we might want to skip updating the bundle file changes, as it never left the experimental stage, and is mainly obsoleted by the addition of stack deploy through compose-files

Hi @thaJeztah,
In doubt i added it in the patch, didn't knew it was obsolete.
Will be removed in the next commit.

silvin-lubecki · 2019-02-11T10:33:23Z

Hello @vdaviot, thank you for opening this PR.
There are still some CI failures:

lint step

cli/command/bundlefile/bundlefile.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/container/formatter_stats.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/container/stats_helpers.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/formatter/formatter.go:1::warning: file is not gofmted with -s (gofmt)
cli/compose/convert/service.go:1::warning: file is not gofmted with -s (gofmt)
cli/compose/types/types.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/bundlefile/bundlefile.go:1::warning: file is not goimported (goimports)
cli/command/container/formatter_stats.go:1::warning: file is not goimported (goimports)
cli/command/container/stats_helpers.go:1::warning: file is not goimported (goimports)
cli/command/formatter/formatter.go:1::warning: file is not goimported (goimports)
cli/compose/convert/service.go:1::warning: file is not goimported (goimports)
cli/compose/types/types.go:1::warning: file is not goimported (goimports)
cli/compose/convert/service.go:217:1:warning: exported function CheckAutoRangeValues should have comment or be unexported (golint)
cli/compose/convert/service.go:249:1:warning: comment on exported function CheckAutoRangeDeclaration should be of the form "CheckAutoRangeDeclaration ..." (golint)
cli/compose/types/types.go:156:6:warning: exported type AutoRange should have comment or be unexported (golint)

Test step

=== FAIL: cli/compose/schema TestValidateCredentialSpecs/3.3 (0.00s)
    --- FAIL: TestValidateCredentialSpecs/3.3 (0.00s)
        schema_test.go:113: assertion failed: expected an error, got nil

=== FAIL: cli/compose/schema TestValidateCredentialSpecs/3.4 (0.01s)
    --- FAIL: TestValidateCredentialSpecs/3.4 (0.01s)
        schema_test.go:113: assertion failed: expected an error, got nil

=== FAIL: cli/compose/schema TestValidateCredentialSpecs/3.5 (0.00s)
    --- FAIL: TestValidateCredentialSpecs/3.5 (0.00s)
        schema_test.go:113: assertion failed: expected an error, got nil

=== FAIL: cli/compose/schema TestValidateCredentialSpecs/3.6 (0.01s)
    --- FAIL: TestValidateCredentialSpecs/3.6 (0.01s)
        schema_test.go:113: assertion failed: expected an error, got nil

=== FAIL: cli/compose/schema TestValidateCredentialSpecs/3.7 (0.00s)
    --- FAIL: TestValidateCredentialSpecs/3.7 (0.00s)
        schema_test.go:113: assertion failed: expected an error, got nil

Validate step:

github.com/docker/swarmkit: Err: exit status 128, out: fatal: reference is not a tree: a85e8e794fea21f8ec83d8369fd5acb9fd7afabe

silvin-lubecki · 2019-02-11T10:37:23Z

cli/command/container/formatter_stats.go

@@ -13,11 +13,13 @@ const (
 	winOSType                  = "windows"
 	defaultStatsTableFormat    = "table {{.ID}}\t{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.NetIO}}\t{{.BlockIO}}\t{{.PIDs}}"
 	winDefaultStatsTableFormat = "table {{.ID}}\t{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}\t{{.BlockIO}}"
+	autoRangeStatsTableFormat  = "table {{.ID}}  {{.MemUsage}}  {{.AutoRange}}"


Shouldn't you use \t between columns?
table {{.ID}}\t{{.MemUsage}}\t{{.AutoRange}}

Hi @silvin-lubecki
Concerning the lint / test / validate steps, i'm correcting those right now, should be commited today.

For the Format, i had some trouble with values not being placed at the right spot, that's why i did it like that, but i can totally change it with \t.

silvin-lubecki · 2019-02-11T10:39:46Z

cli/command/container/formatter_stats.go

@@ -29,6 +31,7 @@ type StatsEntry struct {
 	Container        string
 	Name             string
 	ID               string
+	AutoRange		 string


nit: I think you should configure your text editor/IDE to run a go fmt each time you save a go file.

Will do it ASAP

silvin-lubecki · 2019-02-11T10:43:46Z

cli/command/container/formatter_stats.go


 	containerHeader = "CONTAINER"
 	cpuPercHeader   = "CPU %"
 	netIOHeader     = "NET I/O"
 	blockIOHeader   = "BLOCK I/O"
+	autoRangeHeader = "  Cur MIN/MAX      Opti SOFT/HARD     | CPUS /  %  /  Time"


I'm not familiar with the container stats formatter, but it feels weird having these spaces at the beginning and all these lower cases letters (all other headers are fully upper case).
It also feels like it is 3 different columns( Cur MIN/MAX, Opti SOFT/HARD and CPU / % / Time)? Shouldn't they be 3 different headers? And what is this | for before CPUS? (maybe I'm missing something 😅 )

You are right, should have wrote everything in caps and yes they should, it totally make sense.
The logic behind this was:
I didn't wanted to modify the formatter too heavily implementing new vars that weren't useful for every users, so i did it kinda custom as you see. But the spacing is kinda bad so i should maybe do it differently. Implementing a var for each Headers is a good start, as it should help keep the spacing right

silvin-lubecki · 2019-02-11T10:45:26Z

cli/command/container/stats_helpers.go

+				mem, memLimit          			float64
+				blkRead, blkWrite      			uint64 // Only used on Linux
+				pidsStatsCurrent       			uint64
+				memoryAutoRange, cpuAutoRange   string


nit: This file is missing a go fmt

silvin-lubecki · 2019-02-11T10:48:31Z

cli/command/container/stats_helpers.go

@@ -164,6 +169,65 @@ func collect(ctx context.Context, s *Stats, cli client.APIClient, streamStats bo
 	}
 }

+func formatCPUAutoRange(ar map[string]string) string {
+	numCPU, exist := ar["numCPU"]
+	if !exist {


I guess this pattern can be extracted and factorized 😄

numCPU := getAutoRangeValue(ar, "numCPU") percentOpti := getAutoRangeValue(ar, "percentOpti") ...

silvin-lubecki · 2019-02-11T10:52:01Z

cli/command/container/stats_helpers.go

+func addUnit(value string) string {
+	lval := len(value)
+
+	if lval > 9 {


I think there's already a vendored library for that. Can you check github.com/go-units/size.go ? 🤔

silvin-lubecki · 2019-02-11T10:54:53Z

vendor.conf

@@ -14,7 +14,7 @@ github.com/cpuguy83/go-md2man v1.0.8
 github.com/davecgh/go-spew 346938d642f2ec3594ed81d874461961cd0faa76 # v1.1.0
 github.com/dgrijalva/jwt-go a2c85815a77d0f951e33ba4db5ae93629a1530af
 github.com/docker/distribution 83389a148052d74ac602f5f1d62f86ff2f3c4aa5
-github.com/docker/docker 50e63adf30d33fc1547527a4097c796cbe4b770f
+github.com/docker/docker 9e351d037bfb61a94de4241ced02cfbb7db8e3b3 https://github.com/vdaviot/engine.git


I will add a [WIP] to this PR title as long as the PR is not merged on moby/moby. I think it's not desirable to merge it as is, depending on a personal fork of moby/moby. Same for swarmkit.

@silvin-lubecki yes, of course, thanks

silvin-lubecki · 2019-02-11T10:56:33Z

cli/compose/convert/service.go

+			case "threshold":
+				threshold, _ = strconv.Atoi(v)
+				if threshold <= 0 || threshold > 100 {
+					return fmt.Errorf("Wrong vlaue for threshold")


nit: typo
Maybe fmt.Errorf("invalid threshold %q", v) ?

cli/compose/convert/service.go

silvin-lubecki · 2019-02-11T11:00:04Z

cli/compose/convert/service.go

+
+		// Do checks on values
+		if min > max && max != -1 || min < -1 || max < -1 {
+			return fmt.Errorf("Empty/Wrong values for min or max value")


nit: fmt.Errorf("invalid min %q or max %q values", min, max)

silvin-lubecki · 2019-02-11T17:07:10Z

cli/command/container/formatter_stats.go

@@ -161,6 +191,50 @@ func (c *statsContext) MarshalJSON() ([]byte, error) {
 	return formatter.MarshalJSON(c)
 }

+func (c *statsContext) CurrentMemoryMin() string {
+	if c.s.CurrentMemoryMin == "--" {


I think there is still room for some factorization here 😺

silvin-lubecki · 2019-02-11T17:07:42Z

Thanks @vdaviot for the modifications, that's better 👍

codecov-io · 2019-02-12T09:59:00Z

Codecov Report

Merging #1664 into master will decrease coverage by 0.12%.
The diff coverage is 71.68%.

@@            Coverage Diff             @@
##           master    #1664      +/-   ##
==========================================
- Coverage   56.26%   56.14%   -0.13%     
==========================================
  Files         308      306       -2     
  Lines       21293    21052     -241     
==========================================
- Hits        11980    11819     -161     
+ Misses       8439     8383      -56     
+ Partials      874      850      -24

silvin-lubecki · 2019-02-13T14:47:09Z

@vdaviot your last commit new bindata is really weird, are you sure it's not an error? I think you should revert it.

silvin-lubecki · 2019-02-13T14:47:30Z

And can you squash your commits? 👍

vdaviot · 2019-02-13T15:14:03Z

@silvin-lubecki After looking again, yes it seems weird, but it was generated by
$ go generate github.com/docker/cli/cli/compose/schema
and passed through
$ scripts/validate/check-git-diff cli/compose/schema/bindata.go

Is this the right way to generate a proper bindata.go ?

Anyway i will revert it.
Commits will be squashed when the code pass all the checks (if it's ok, otherwise i'll do it first)

silvin-lubecki · 2019-02-13T15:22:24Z

There is a target in the Makefile to update the schemas:
https://github.com/docker/cli/blob/master/Makefile#L91

But for your case I think you don't use the same esc binary version as pined in the Dockerfile:
https://github.com/docker/cli/blob/master/dockerfiles/Dockerfile.dev#L12

silvin-lubecki · 2019-02-13T15:22:51Z

And yes you're right about the squash, take it as a reminder then 😄

Signed-off-by: Philipp Schmied <pschmied@schutzwerk.com>

Signed-off-by: Michael Käufl <docker@c.michael-kaeufl.de>

Signed-off-by: Valentin Daviot <valentin.daviot@alterway.fr> Signed-off-by: Valentin Daviot <valentin.daviot@alterway.fr>

vdaviot requested a review from silvin-lubecki as a code owner February 8, 2019 16:18

GordonTheTurtle added the dco/no label Feb 8, 2019

GordonTheTurtle added the status/0-triage label Feb 8, 2019

This was referenced Feb 8, 2019

# [API] AutoRange type moby/swarmkit#2818

Open

# [FEATURE] AutoRange docker/engine#148

Closed

vdaviot force-pushed the master branch 2 times, most recently from a0445ee to 0035956 Compare February 8, 2019 16:30

GordonTheTurtle removed the dco/no label Feb 8, 2019

thaJeztah reviewed Feb 8, 2019

View reviewed changes

vdaviot mentioned this pull request Feb 11, 2019

[FEATURE] AutoRange moby/moby#38706

Open

silvin-lubecki reviewed Feb 11, 2019

View reviewed changes

cli/compose/convert/service.go Outdated Show resolved Hide resolved

silvin-lubecki reviewed Feb 11, 2019

View reviewed changes

silvin-lubecki changed the title ~~# [FEATURE] AutoRange parsing & format~~ [WIP][FEATURE] AutoRange parsing & format Feb 11, 2019

silvin-lubecki reviewed Feb 11, 2019

View reviewed changes

GordonTheTurtle added the dco/no label Feb 12, 2019

sw-pschmied and others added 2 commits March 27, 2019 14:25

Prevent overwriting irregular files (cp, save, export commands)

3867359

Signed-off-by: Philipp Schmied <pschmied@schutzwerk.com>

Fix typos

12e86d0

Signed-off-by: Michael Käufl <docker@c.michael-kaeufl.de>

vdaviot force-pushed the master branch from f465e74 to b793355 Compare March 27, 2019 14:01

GordonTheTurtle removed the dco/no label Mar 27, 2019

vdaviot force-pushed the master branch from b793355 to c29cd6c Compare March 27, 2019 14:02

autorange feature. code passed through gofmt / golint / goimports

c48982d

Signed-off-by: Valentin Daviot <valentin.daviot@alterway.fr> Signed-off-by: Valentin Daviot <valentin.daviot@alterway.fr>

vdaviot force-pushed the master branch from c29cd6c to c48982d Compare March 27, 2019 14:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP][FEATURE] AutoRange parsing & format #1664

[WIP][FEATURE] AutoRange parsing & format #1664

vdaviot commented Feb 8, 2019 •

edited

GordonTheTurtle commented Feb 8, 2019

thaJeztah Feb 8, 2019

vdaviot Feb 11, 2019

silvin-lubecki commented Feb 11, 2019

silvin-lubecki Feb 11, 2019

vdaviot Feb 11, 2019 •

edited

silvin-lubecki Feb 11, 2019

vdaviot Feb 11, 2019

silvin-lubecki Feb 11, 2019

vdaviot Feb 11, 2019

silvin-lubecki Feb 11, 2019

silvin-lubecki Feb 11, 2019

silvin-lubecki Feb 11, 2019

silvin-lubecki Feb 11, 2019

vdaviot Feb 11, 2019

silvin-lubecki Feb 11, 2019

silvin-lubecki Feb 11, 2019

silvin-lubecki Feb 11, 2019

silvin-lubecki commented Feb 11, 2019

codecov-io commented Feb 12, 2019 •

edited

silvin-lubecki commented Feb 13, 2019

silvin-lubecki commented Feb 13, 2019

vdaviot commented Feb 13, 2019

silvin-lubecki commented Feb 13, 2019

silvin-lubecki commented Feb 13, 2019

[WIP][FEATURE] AutoRange parsing & format #1664

Are you sure you want to change the base?

[WIP][FEATURE] AutoRange parsing & format #1664

Conversation

vdaviot commented Feb 8, 2019 • edited

Cute animal

GordonTheTurtle commented Feb 8, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

silvin-lubecki commented Feb 11, 2019

Choose a reason for hiding this comment

vdaviot Feb 11, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

silvin-lubecki commented Feb 11, 2019

codecov-io commented Feb 12, 2019 • edited

Codecov Report

silvin-lubecki commented Feb 13, 2019

silvin-lubecki commented Feb 13, 2019

vdaviot commented Feb 13, 2019

silvin-lubecki commented Feb 13, 2019

silvin-lubecki commented Feb 13, 2019

vdaviot commented Feb 8, 2019 •

edited

vdaviot Feb 11, 2019 •

edited

codecov-io commented Feb 12, 2019 •

edited