support for quota, allocator, maintenance, master, and agent v1.2.x APIs #312

jdef · 2017-07-25T14:23:46Z

updated 1.2.x protobufs
refactored client, encoding, httpcli, recordio packages to account for v1 protocol changes in 1.2.x
updated and enhanced examples

fixes support new content and message type headers w/ mesos 1.2.0 #300, add support for master 1.2.x operator API #310

jdef · 2017-07-25T14:25:16Z

make test fails for me because of some GoString related test. Suspect a bug in gogo/protobuf. I hacked the test to output the result of the GoString func that failed to parse:

--- FAIL: TestEvent_TaskUpdatedGoString (0.00s)
        masterpb_test.go:7963: failed to parse &master.Event_TaskUpdated{FrameworkID: mesos.FrameworkID{Value: "CW0R6JWXGjhG3m3pAnAbJV0jG7DTCYQvCekLB5zY0FxUZa6N8JzsTMxQyffu",
                },
                Status: mesos.TaskStatus{TaskID: mesos.TaskID{Value: "0g2LMJZn5Nmp9rhboWMBvMfvJDMEGJ0q4UpS5tjBVCB3FuDclnFoTG3HsJH57ufYnZ3nQzz7COA4",
                },
                State: func(v mesos.TaskState) *mesos.TaskState { return &v } ( 5 ),
                Message: func(v string) *string { return &v } ( "6G44Yqd" ),
                Source: func(v mesos.TaskStatus_Source) *mesos.TaskStatus_Source { return &v } ( 1 ),
                Reason: func(v mesos.TaskStatus_Reason) *mesos.TaskStatus_Reason { return &v } ( 20 ),
                ExecutorID: &mesos.ExecutorID{Value: "FlRaWodJlx61RG5k3NjbS3bOFGFKiqvR4R8yWOdRMkgJNmbQ9BhtGXEIHOw",
                },
                Timestamp: func(v float64) *float64 { return &v } ( 0.1795823327242275 ),
                Healthy: func(v bool) *bool { return &v } ( false ),
                CheckStatus: &mesos.CheckStatusInfo{Type: func(v mesos.CheckInfo_Type) *mesos.CheckInfo_Type { return &v } ( 2 ),
                Command: &mesos.CheckStatusInfo_Command{ExitCode: func(v int32) *int32 { return &v } ( -69695686 ),
                },
                HTTP: &mesos.CheckStatusInfo_Http{StatusCode: func(v uint32) *uint32 { return &v } ( 0x95f985dc ),
                },
                },
                Labels: &mesos.Labels{Labels: []mesos.Label{mesos.Label{Key:"tBGAfISwYn3XJusiJitNyQSZlnTBg", Value:(*string)(0xc42015c600)}, mesos.Label{Key:"EiJHST1g4QWQeYDPGaCHmsPtRzq27ZMznvMLGfoQd20c9oLC5i2UqDi
mc8oPnI3nhWFvzK1ui", Value:(*string)(0xc42015c620)}, mesos.Label{Key:"qPSxKCnx1UW6uSjUkeaBkK", Value:(*string)(0xc42015c630)}, mesos.Label{Key:"tydcu0owgtFNChnDhlO95bjonwijuOkcqsBY2ZmS0rWcLdQNce8D4qWompMDeHrStSkgC
JpBKEa", Value:(*string)(0xc42015c640)}},
                },
                UnreachableTime: &mesos.TimeInfo{Nanoseconds: -5095539703890242531,
                },
                },
                State: func(v master.mesos.TaskState) *master.mesos.TaskState { return &v } ( 1 ),
                }
panic: 24:8: expected expression (and 3 more errors) [recovered]   
        panic: 24:8: expected expression (and 3 more errors)

goroutine 341 [running]:
testing.tRunner.func1(0xc42014d6c0)
        /usr/local/go-1.8.1/src/testing/testing.go:622 +0x55f
panic(0xc50620, 0xc4203ff320)
        /usr/local/go-1.8.1/src/runtime/panic.go:489 +0x2f0
github.com/mesos/mesos-go/api/v1/lib/master.TestEvent_TaskUpdatedGoString(0xc42014d6c0)
        /home/vagrant/mesos-go-workspace/src/github.com/mesos/mesos-go/api/v1/lib/master/masterpb_test.go:7964 +0x517
testing.tRunner(0xc42014d6c0, 0xcb4db0)
        /usr/local/go-1.8.1/src/testing/testing.go:657 +0x108
created by testing.(*T).Run
        /usr/local/go-1.8.1/src/testing/testing.go:697 +0x544
FAIL    github.com/mesos/mesos-go/api/v1/lib/master     4.348s

The (generated, then hacked) test case is:

func TestEvent_TaskUpdatedGoString(t *testing.T) {
        popr := math_rand.New(math_rand.NewSource(time.Now().UnixNano()))
        p := NewPopulatedEvent_TaskUpdated(popr, false)
        s1 := p.GoString()
        s2 := fmt.Sprintf("%#v", p)
        if s1 != s2 {
                t.Fatalf("GoString want %v got %v", s1, s2)
        }
        _, err := go_parser.ParseExpr(s1)
        if err != nil {
                t.Errorf("failed to parse %s", s1) // I added this to see what GoString was returning
                panic(err) // fails the test
        }
}

coveralls · 2017-07-25T17:59:22Z

Coverage remained the same at 52.609% when pulling 179cbce on jdef_master_v1_api_support into 464c8d6 on master.

coveralls · 2017-07-27T04:59:54Z

Coverage remained the same at 52.609% when pulling 5df56f4 on jdef_master_v1_api_support into 464c8d6 on master.

coveralls · 2017-07-31T10:41:43Z

Coverage remained the same at 52.609% when pulling c82e1af on jdef_master_v1_api_support into 464c8d6 on master.

… handle Mesos v1 protocol changes introduced in Mesos 1.2.x

…erator APIs

coveralls · 2017-08-30T15:57:36Z

Coverage decreased (-2.0%) to 50.618% when pulling 6b61399 on jdef_master_v1_api_support into 464c8d6 on master.

jdef · 2017-08-30T20:20:42Z

I've been testing the "attach container" API with an interactive bash shell. Couple of problems observed so far:

when running msh without the -pod flag bash generates warnings at startup

2017/08/30 20:18:57 status update from agent "4849630d-c04b-4ae5-9659-9f40aeab5fd2-S0": TASK_RUNNING
2017/08/30 20:18:57 attaching for interactive session to agent "4849630d-c04b-4ae5-9659-9f40aeab5fd2-S0" container "d340b90d-d733-4c67-b909-67a0488759e7"
original window size is 213 x 48
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

intermittent task (I/O switchboard) failures when attaching to the container (several related logs follow, ignore the tiimestamp skew - logs were sampled from several test runs. the errors are the same across failed runs)

2017/08/30 20:15:05 status update from agent "4849630d-c04b-4ae5-9659-9f40aeab5fd2-S1": TASK_RUNNING
2017/08/30 20:15:05 attaching for interactive session to agent "4849630d-c04b-4ae5-9659-9f40aeab5fd2-S1" container "9f9d833f-ff32-4943-a397-f2eba78b4448"
original window size is 213 x 48
2017/08/30 20:15:05 attached input stream error Post http://10.2.0.5:5051/api/v1: context canceled
2017/08/30 20:15:05 Exiting because task 20170830T201505Z is in an unexpected state TASK_FAILED with reason REASON_COMMAND_EXECUTOR_FAILED from source SOURCE_EXECUTOR with message 'Command terminated with signal Window changed'
2017/08/30 20:15:05 exit code 3

slave logs:

E0830 18:46:35.124127  2119 switchboard.cpp:902] Unexpected termination of I/O switchboard server: 'IOSwitchboard' exited with status 1 for container 24a90359-b0e4-4c0a-9051-e6552811ae22.9ad1023b-cb7a-45fc-8053-dc
a0e888c9d3

container stderr:

$ cat /var/lib/mesos/slaves/4849630d-c04b-4ae5-9659-9f40aeab5fd2-S1/frameworks/93c06854-b91a-4f8d-86b4-cdf1cc628613-0101/executors/msh_20170830T184634Z/runs/latest/containers/9ad1023b-cb7a-45fc-8053-dca0e888c9d3/stderr 
The io switchboard server failed: Failed redirecting stdout: Input/output error

container stdout:

$ cat /var/lib/mesos/slaves/4849630d-c04b-4ae5-9659-9f40aeab5fd2-S1/frameworks/93c06854-b91a-4f8d-86b4-cdf1cc628613-0101/executors/msh_20170830T184634Z/runs/latest/containers/9ad1023b-cb7a-45fc-8053-dca0e888c9d3/stdout
Executing pre-exec command '{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/usr\/libexec\/mesos\/mesos-containerizer"}'

There are (currently) a number of open MESOS tickets related to container attachment: https://issues.apache.org/jira/browse/MESOS-7103

coveralls · 2017-08-30T20:27:28Z

Coverage decreased (-2.6%) to 50.048% when pulling 8cd48c5 on jdef_master_v1_api_support into 464c8d6 on master.

jdef · 2017-09-06T17:46:10Z

api/v1/lib/httpcli/http.go

@@ -27,7 +30,8 @@ type ProtocolError string
 func (pe ProtocolError) Error() string { return string(pe) }

 const (
-	debug = false // TODO(jdef) kill me at some point
+	debug             = false // TODO(jdef) kill me at some point
+	MediaTypeRecordIO = encoding.MediaType("application/recordio")


why is this public?

jdef · 2017-09-06T17:47:47Z

api/v1/lib/httpcli/http.go

+		enc    = c.codec.NewEncoder(func() framing.Writer { return recordio.NewWriter(pw) })
+	)
+
+	req, err := http.NewRequest("POST", c.url, pr)


move to var block?

no, because err is already declared

jdef · 2017-09-06T17:49:41Z

api/v1/lib/httpcli/http.go

+		}
+
+	case http.StatusAccepted:
+		// nothing to validate, we're not expecting any response entity in this case


maybe a TODO to validate ContentLength (if Mesos actually supplies that?)

coveralls · 2017-09-10T11:58:11Z

Coverage increased (+0.4%) to 52.989% when pulling 1bedc2a on jdef_master_v1_api_support into 464c8d6 on master.

coveralls · 2017-09-10T11:58:11Z

Coverage decreased (-0.08%) to 52.527% when pulling 1bedc2a on jdef_master_v1_api_support into 464c8d6 on master.

coveralls · 2017-09-10T15:33:31Z

Coverage decreased (-0.008%) to 52.602% when pulling a088793 on jdef_master_v1_api_support into 464c8d6 on master.

coveralls · 2017-09-10T20:48:34Z

Coverage increased (+1.3%) to 53.897% when pulling 848347f on jdef_master_v1_api_support into 464c8d6 on master.

coveralls · 2017-09-10T21:31:11Z

Coverage increased (+1.7%) to 54.172% when pulling 4c21684 on jdef_master_v1_api_support into 479884c on master.

coveralls · 2017-09-11T15:36:07Z

Coverage increased (+1.7%) to 54.172% when pulling 14d08b2 on jdef_master_v1_api_support into 479884c on master.

coveralls · 2017-09-11T16:38:42Z

Coverage increased (+3.2%) to 55.718% when pulling 2a8c4c4 on jdef_master_v1_api_support into 479884c on master.

coveralls · 2017-09-11T17:44:48Z

Coverage increased (+3.4%) to 55.887% when pulling 6329032 on jdef_master_v1_api_support into 479884c on master.

coveralls · 2017-09-11T20:38:37Z

Coverage increased (+4.6%) to 57.067% when pulling 93a0891 on jdef_master_v1_api_support into 479884c on master.

coveralls · 2017-09-11T21:48:11Z

Coverage increased (+4.6%) to 57.067% when pulling e6802a2 on jdef_master_v1_api_support into 479884c on master.

James DeFelice added 4 commits July 25, 2017 13:32

proto/build: initial support for v1 quota API

8a2ea68

proto/build: initial support for v1 allocator API

ff71cb9

proto/build: initial support for v1 maintenance API

e0d431a

proto/build: initial support for v1 master API

0912935

jdef changed the title ~~[WIP] protos for quota, allocator, maintenance, and master v1 APIs~~ [WIP] protos for quota, allocator, maintenance, and master v1.2.x APIs Jul 25, 2017

jdef mentioned this pull request Jul 25, 2017

GoString is generating unparseable golang gogo/protobuf#312

Closed

proto: set gostringer_all=false to avoid broken generated tests

179cbce

James DeFelice added 3 commits July 27, 2017 04:39

fix: proto/lib: ExecutorInfo.Command is optional, not required

9baddaa

fix: examples: ExecutorInfo.Command is optional, not required

c09e0b3

examples: msh supports -tty and -pod flags

5df56f4

James DeFelice added 2 commits July 29, 2017 23:33

vendor: bump gogo/protobuf to latest for gostring fix

fae354a

proto: re-enable gostring option (pick up fixes from latest gogo)

c82e1af

James DeFelice added 2 commits August 30, 2017 15:45

refactor: upgrade client, encoding, httpcli, and recordio packages to…

0c61722

… handle Mesos v1 protocol changes introduced in Mesos 1.2.x

msh: flesh out -interactive and -tty support using new Mesos 1.2.x op…

6b61399

…erator APIs

jdef changed the title ~~[WIP] protos for quota, allocator, maintenance, and master v1.2.x APIs~~ [WIP] support for quota, allocator, maintenance, master, and agent v1.2.x APIs Aug 30, 2017

James DeFelice added 2 commits August 30, 2017 20:14

agent: AttachContainerXxx call helpers

784e99f

msh: cleanup tty and attach-container code

8cd48c5

jdef commented Sep 6, 2017

View reviewed changes

James DeFelice added 5 commits September 6, 2017 17:54

agent/calls: implement API helpers for remaining calls

deec2a8

msh: fix go vet errors

a98de55

httpcli: address review feedback

3ca66ef

make: add INSTALL_FLAGS for customizing binary generation

4846eff

agent/calls: introduce Sender and Request for agent clients

094a1aa

agent/calls: sanity-check Example, nothing should blow up

1bedc2a

gen: sender/httpsender refactoring/cleanup

a088793

gen: sender unit test and fixes

4c21684

jdef force-pushed the jdef_master_v1_api_support branch from 848347f to 4c21684 Compare September 10, 2017 21:12

gen: fix vet errors in sender

14d08b2

gen: refactor + unit test for httpsender

6329032

jdef force-pushed the jdef_master_v1_api_support branch from 2a8c4c4 to 6329032 Compare September 11, 2017 17:23

James DeFelice added 2 commits September 11, 2017 19:49

lib/{http,recordio}: more unit tests

c5d4a73

encoding: API cleanup

93a0891

This was referenced Sep 11, 2017

add api/v1/lib/scheduler/calls Sender API #316

Open

add api/v1/lib/executor/calls Sender API #317

Closed

add api/v1/lib/httpcli/httpsched NewSender API #318

Open

jdef changed the title ~~[WIP] support for quota, allocator, maintenance, master, and agent v1.2.x APIs~~ support for quota, allocator, maintenance, master, and agent v1.2.x APIs Sep 11, 2017

jdef added api/v1 class/enhancement PTAL labels Sep 11, 2017

updated changelog w/ summary for #312

e6802a2

jdef merged commit f0f18ff into master Sep 11, 2017

jdef deleted the jdef_master_v1_api_support branch September 11, 2017 21:29

jdef mentioned this pull request Sep 12, 2017

add support for master 1.2.x operator API #310

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support for quota, allocator, maintenance, master, and agent v1.2.x APIs #312

support for quota, allocator, maintenance, master, and agent v1.2.x APIs #312

jdef commented Jul 25, 2017 •

edited

Loading

jdef commented Jul 25, 2017 •

edited

Loading

coveralls commented Jul 25, 2017 •

edited

Loading

coveralls commented Jul 27, 2017 •

edited

Loading

coveralls commented Jul 31, 2017 •

edited

Loading

coveralls commented Aug 30, 2017 •

edited

Loading

jdef commented Aug 30, 2017 •

edited

Loading

coveralls commented Aug 30, 2017 •

edited

Loading

jdef Sep 6, 2017

jdef Sep 6, 2017

jdef Sep 6, 2017

jdef Sep 6, 2017

coveralls commented Sep 10, 2017

coveralls commented Sep 10, 2017

coveralls commented Sep 10, 2017 •

edited

Loading

coveralls commented Sep 10, 2017 •

edited

Loading

coveralls commented Sep 10, 2017 •

edited

Loading

coveralls commented Sep 11, 2017 •

edited

Loading

coveralls commented Sep 11, 2017 •

edited

Loading

coveralls commented Sep 11, 2017 •

edited

Loading

coveralls commented Sep 11, 2017 •

edited

Loading

coveralls commented Sep 11, 2017 •

edited

Loading

support for quota, allocator, maintenance, master, and agent v1.2.x APIs #312

support for quota, allocator, maintenance, master, and agent v1.2.x APIs #312

Conversation

jdef commented Jul 25, 2017 • edited Loading

jdef commented Jul 25, 2017 • edited Loading

coveralls commented Jul 25, 2017 • edited Loading

coveralls commented Jul 27, 2017 • edited Loading

coveralls commented Jul 31, 2017 • edited Loading

coveralls commented Aug 30, 2017 • edited Loading

jdef commented Aug 30, 2017 • edited Loading

coveralls commented Aug 30, 2017 • edited Loading

jdef Sep 6, 2017

Choose a reason for hiding this comment

jdef Sep 6, 2017

Choose a reason for hiding this comment

jdef Sep 6, 2017

Choose a reason for hiding this comment

jdef Sep 6, 2017

Choose a reason for hiding this comment

coveralls commented Sep 10, 2017

coveralls commented Sep 10, 2017

coveralls commented Sep 10, 2017 • edited Loading

coveralls commented Sep 10, 2017 • edited Loading

coveralls commented Sep 10, 2017 • edited Loading

coveralls commented Sep 11, 2017 • edited Loading

coveralls commented Sep 11, 2017 • edited Loading

coveralls commented Sep 11, 2017 • edited Loading

coveralls commented Sep 11, 2017 • edited Loading

coveralls commented Sep 11, 2017 • edited Loading

jdef commented Jul 25, 2017 •

edited

Loading

jdef commented Jul 25, 2017 •

edited

Loading

coveralls commented Jul 25, 2017 •

edited

Loading

coveralls commented Jul 27, 2017 •

edited

Loading

coveralls commented Jul 31, 2017 •

edited

Loading

coveralls commented Aug 30, 2017 •

edited

Loading

jdef commented Aug 30, 2017 •

edited

Loading

coveralls commented Aug 30, 2017 •

edited

Loading

coveralls commented Sep 10, 2017 •

edited

Loading

coveralls commented Sep 10, 2017 •

edited

Loading

coveralls commented Sep 10, 2017 •

edited

Loading

coveralls commented Sep 11, 2017 •

edited

Loading

coveralls commented Sep 11, 2017 •

edited

Loading

coveralls commented Sep 11, 2017 •

edited

Loading

coveralls commented Sep 11, 2017 •

edited

Loading

coveralls commented Sep 11, 2017 •

edited

Loading