Agent introspection command improvements #7465

Merged
merged 1 commit into from Jun 8, 2017

Conversation

Projects
None yet
4 participants
Member

axw commented Jun 7, 2017

Description of change

This branch brings several improvements to the
agent introspection shell functions:

  • we no longer use socat to communicate with the
    introspection socket, which was causing problems
    with the CPU profile endpoint. Instead, we add
    a new "juju-introspect" command/symlink to jujud,
    which serves as a frontend to the introspection
    socket. The helpers for getting specific profiles
    are left as bash functions.
  • the new juju-introspect command has a --listen
    flag which, if specified, causes the command to
    expose the socket over HTTP. This enables the
    user to use "go tool pprof" with the HTTP endpoint,
    for example.
  • A new function, juju-cpu-profile, is added, which
    will capture a CPU profile. The first (optional)
    argument is interpreted as the number of seconds
    to sample for.

Since we are no longer writing out the raw HTTP output,
the output of the various juju-foo-profile functions
no longer include the HTTP response header, so the
output can be fed directly into "go tool pprof".

By moving the main frontend out of the bash script and
into Go code, we get a step closer to having
introspection on Windows. We still need an alternative
to the abstract domain sockets solution that we have
today (probably named pipes?)

QA steps

  1. juju bootstrap
  2. juju deploy -m controller ubuntu --to 0
  3. juju ssh -m controller 0
  4. juju-goroutines
  5. juju-goroutines unit-ubuntu-0
  6. juju-heap-profile
  7. juju-cpu-profile # samples for 30 seconds
  8. juju-cpu-profile 5 # samples for 5 seconds
  9. juju-introspect --listen=:6060
    (then on another machine, go tool pprof path/to/jujud http::6060)
  10. juju-introspect metrics # scrapes prometheus metrics

Documentation changes

We should update any debugging docs we have with mention of the new "juju-cpu-profile" helper, and possibly the "juju-introspect --listen=..." command.

Bug reference

None.

This changes the actual output of the scripts. So my personal bias is to probably put it into 2.2.0 rather than waiting for 2.2.1 and having a user-visible compatibility change in a point release. (people using <THIS have to munge the output to make it work with other tooling, and would break if they munge after this.)

cmd/jujud/agent/machine_test.go
@@ -800,7 +800,7 @@ func (s *MachineSuite) TestMachineAgentSymlinks(c *gc.C) {
_, done := s.waitForOpenState(c, a)
// Symlinks should have been created
- for _, link := range []string{jujuRun, jujuDumpLogs} {
+ for _, link := range []string{jujuRun, jujuDumpLogs, jujuIntrospect} {
@jameinel

jameinel Jun 7, 2017

Owner

should this be 'jujudSymlinks' ?

@axw

axw Jun 7, 2017

Member

Fixed, thanks.

cmd/jujud/agent/machine_test.go
@@ -820,7 +820,7 @@ func (s *MachineSuite) TestMachineAgentSymlinkJujuRunExists(c *gc.C) {
defer a.Stop()
// Pre-create the symlinks, but pointing to the incorrect location.
- links := []string{jujuRun, jujuDumpLogs}
+ links := []string{jujuRun, jujuDumpLogs, jujuIntrospect}
@jameinel

jameinel Jun 7, 2017

Owner

and this

@axw

axw Jun 7, 2017

Member

Fixed, thanks.

worker/introspection: improve shell functions
This branch brings several improvements to the
agent introspection shell functions:

- we no longer use socat to communicate with the
  introspection socket, which was causing problems
  with the CPU profile endpoint. Instead, we add
  a new "juju-introspect" command/symlink to jujud,
  which serves as a frontend to the introspection
  socket. The helpers for getting specific profiles
  are left as bash functions.
- the new juju-introspect command has a --listen
  flag which, if specified, causes the command to
  expose the socket over HTTP. This enables the
  user to use "go tool pprof" with the HTTP endpoint,
  for example.
- A new function, juju-cpu-profile, is added, which
  will capture a CPU profile. The first (optional)
  argument is interpreted as the number of seconds
  to sample for.

Since we are no longer writing out the raw HTTP output,
the output of the various juju-foo-profile functions
no longer include the HTTP response header, so the
output can be fed directly into "go tool pprof".

By moving the main frontend out of the bash script and
into Go code, we get a step closer to having
introspection on Windows. We still need an alternative
to the abstract domain sockets solution that we have
today (probably named pipes?)
Member

axw commented Jun 7, 2017

!!doesthisworkatleast!!

Member

axw commented Jun 7, 2017

$$covfefe$$

Contributor

jujubot commented Jun 7, 2017

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

@jujubot jujubot merged commit b2aa2af into juju:2.2 Jun 8, 2017

1 check passed

github-check-merge-juju Ran tests against PR. Use !!.*!! to request another build. IE, !!build!!, !!retry!!
Details
+const introspectCommandDoc = `
+Introspect Juju agents running on this machine.
+
+The juju-introspect can be used to expose the
@wallyworld

wallyworld Jun 8, 2017

Owner

missing "command" in text

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment