Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql: don't bind a Context in BoundAccount #10478

Merged
merged 3 commits into from Mar 9, 2017

Conversation

andreimatei
Copy link
Contributor

@andreimatei andreimatei commented Nov 5, 2016

A BoundAccount is a MemoryAccount bound to a MemoryMonitor. When
creating a BoundAcc, we used to bind a context to be used by the Acc
throughout its lifetime. This means that the context had to stay "valid"
throughout (e.g. the span in the ctx, if any, had to stay open). This is
conceptually a problem because it means you can't open an acc in one
place and close it in a completely different place (and also diminishes
the utility of contexts, as Account operations were not logged in a
context corresponding to the function actually performing them). In
particular, this was a problem for accounts created for RowContainers
and the way how EXPLAIN(TRACE) temporarily hijacks the session's
context: a container's account could easily be bound to one of these
temporarily hijacked contexts, and closed after the corresponding span
had already been Finish()ed.

This patch plumbs through a Context to the accounts, through various
cascading layers.

A bunch of accounts are opened by various planNodes, and there the
question of what context to use I believe is not yet flushed out. So
far, planNodes that already used contexts were using the planner's
current context (which is the session's context). Not all planNodes
maintained a reference to the planner, though. Where a reference
existed, I continued using it. Where it didn't, instead of adding one,
I've bound a context directly to the planNode at ctor time.

cc @RaduBerinde in case you've got any thoughts on how planNodes should think about contexts.


This change is Reviewable

@tamird
Copy link
Contributor

tamird commented Nov 5, 2016

Reviewed 17 of 17 files at r1.
Review status: all files reviewed at latest revision, 6 unresolved discussions, some commit checks pending.


pkg/sql/explain.go, line 149 at r1 (raw file):

type explainTypesNode struct {
  ctx      context.Context

a lot of work has gone into unwinding this pattern (of storing a context in structures) from the core. can you explain (in a comment) why it is necessary here? AFAICT, it's to avoid plumbing a context into Start, which would deserve a TODO.


pkg/sql/explain.go, line 257 at r1 (raw file):

type explainPlanNode struct {
  verbose bool
  ctx     context.Context

ditto


pkg/sql/group.go, line 174 at r1 (raw file):

type groupNode struct {
  planner *planner
  ctx     context.Context

ditto


pkg/sql/table_join.go, line 44 at r1 (raw file):

  joinType joinType

  ctx context.Context

ditto


pkg/sql/values.go, line 35 at r1 (raw file):

type valuesNode struct {
  n        *parser.ValuesClause
  ctx      context.Context

ditto


pkg/sql/mon/mem_usage.go, line 456 at r1 (raw file):

  return BoundAccount{
      MemoryAccount: MemoryAccount{curAllocated: capacity},
      mon:           nil,

nit: you could remove this


Comments from Reviewable

@andreimatei
Copy link
Contributor Author

Review status: all files reviewed at latest revision, 6 unresolved discussions, some commit checks failed.


pkg/sql/explain.go, line 149 at r1 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

a lot of work has gone into unwinding this pattern (of storing a context in structures) from the core. can you explain (in a comment) why it is necessary here? AFAICT, it's to avoid plumbing a context into Start, which would deserve a TODO.

Did you read the commit message? The `planNodes` don't generally take contexts for their operations (notably, for `Next`). They use the session's context, through the planner. What I'm doing here is just making that more explicit.

I don't know what the proper thing to in planNodes is. Next might be the one place where it's too expensive to continuously pass contexts through.

What I'm doing here is not quite good, though, because we might want to make the session's context dynamic. But I don't like nodes getting references to the planner just because they need that context. I was thinking of introducing a DynamicCtx, which would be a *Context.


Comments from Reviewable

@knz
Copy link
Contributor

knz commented Nov 5, 2016

On the one hand I support the motion of being more explicit about contexts, however the way you did it looks messy.

See my comments below. I recommend removing the session.Ctx() and planner.ctx() methods entirely and replace them with two methods planner.SessionCtx() and planner.TxnCtx() that retrieve session.context and session.TxnState.Ctx respectively. After searching I have not found a single useful need of the conditional currently present in session.Ctx.

nit: I believe it's "fleshed out" not "flushed out."


Reviewed 17 of 17 files at r1.
Review status: all files reviewed at latest revision, 102 unresolved discussions, some commit checks failed.


pkg/server/admin.go, line 190 at r1 (raw file):

  defer session.Finish(s.server.sqlExecutor)
  r := s.server.sqlExecutor.ExecuteStatements(session, "SHOW DATABASES;", nil)
  defer r.Close(ctx)

probably r.Close(session.context)


pkg/server/admin.go, line 224 at r1 (raw file):

  query := fmt.Sprintf("SHOW GRANTS ON DATABASE %s; SHOW TABLES FROM %s;", escDBName, escDBName)
  r := s.server.sqlExecutor.ExecuteStatements(session, query, nil)
  defer r.Close(ctx)

ditto


pkg/server/admin.go, line 276 at r1 (raw file):

  // Query the descriptor ID and zone configuration for this database.
  {
      path, err := s.queryDescriptorIDPath(ctx, session, []string{req.Database})

not sure this is needed, you can pull it from session


pkg/server/admin.go, line 282 at r1 (raw file):

      resp.DescriptorID = int64(path[1])

      id, zone, zoneExists, err := s.queryZonePath(ctx, session, path)

ditto


pkg/server/admin.go, line 320 at r1 (raw file):

      escQualTable, escQualTable, escQualTable, escQualTable)
  r := s.server.sqlExecutor.ExecuteStatements(session, query, nil)
  defer r.Close(ctx)

ditto


pkg/server/admin.go, line 481 at r1 (raw file):

      // Query the descriptor ID and zone configuration for this table.
      {
          path, err := s.queryDescriptorIDPath(ctx, session, []string{req.Database, req.Table})

ditto


pkg/server/admin.go, line 487 at r1 (raw file):

          resp.DescriptorID = int64(path[2])

          id, zone, zoneExists, err := s.queryZonePath(ctx, session, path)

ditto


pkg/server/admin.go, line 634 at r1 (raw file):

  query := "SELECT username FROM system.users"
  r := s.server.sqlExecutor.ExecuteStatements(session, query, nil)
  defer r.Close(ctx)

ditto


pkg/server/admin.go, line 676 at r1 (raw file):

  }
  r := s.server.sqlExecutor.ExecuteStatements(session, q.String(), q.QueryArguments())
  defer r.Close(ctx)

ditto


pkg/server/admin.go, line 716 at r1 (raw file):

// that are not found will not be returned.
func (s *adminServer) getUIData(
  ctx context.Context, session *sql.Session, user string, keys []string,

arg not needed - session.context should do


pkg/server/admin.go, line 736 at r1 (raw file):

  }
  r := s.server.sqlExecutor.ExecuteStatements(session, query.String(), query.QueryArguments())
  defer r.Close(ctx)

see above


pkg/server/admin.go, line 783 at r1 (raw file):

      // avoid long-running transactions and possible deadlocks.
      br := s.server.sqlExecutor.ExecuteStatements(session, "BEGIN;", nil)
      defer br.Close(ctx)

see above


pkg/server/admin.go, line 789 at r1 (raw file):

      // See if the key already exists.
      resp, err := s.getUIData(ctx, session, s.getUser(req), []string{key})

see above


pkg/server/admin.go, line 802 at r1 (raw file):

          qargs.SetValue(`2`, parser.NewDString(key))
          r := s.server.sqlExecutor.ExecuteStatements(session, query, qargs)
          defer r.Close(ctx)

see above


pkg/server/admin.go, line 815 at r1 (raw file):

          qargs.SetValue(`2`, parser.NewDBytes(parser.DBytes(val)))
          r := s.server.sqlExecutor.ExecuteStatements(session, query, qargs)
          defer r.Close(ctx)

see above


pkg/server/admin.go, line 845 at r1 (raw file):

  }

  resp, err := s.getUIData(ctx, session, s.getUser(req), req.Keys)

see above


pkg/server/admin.go, line 1319 at r1 (raw file):

// if it exists.
func (s *adminServer) queryZone(
  ctx context.Context, session *sql.Session, id sqlbase.ID,

see above


pkg/server/admin.go, line 1325 at r1 (raw file):

  params.SetValue(`1`, parser.NewDInt(parser.DInt(id)))
  r := s.server.sqlExecutor.ExecuteStatements(session, query, params)
  defer r.Close(ctx)

see above


pkg/server/admin.go, line 1353 at r1 (raw file):

// ZoneConfig specified for the object IDs in the path.
func (s *adminServer) queryZonePath(
  ctx context.Context, session *sql.Session, path []sqlbase.ID,

see above


pkg/server/admin.go, line 1367 at r1 (raw file):

// parent ID.
func (s *adminServer) queryNamespaceID(
  ctx context.Context, session *sql.Session, parentID sqlbase.ID, name string,

see above


pkg/server/admin.go, line 1374 at r1 (raw file):

  params.SetValue(`2`, parser.NewDString(name))
  r := s.server.sqlExecutor.ExecuteStatements(session, query, params)
  defer r.Close(ctx)

see above


pkg/server/admin.go, line 1399 at r1 (raw file):

// databases ID, and the table ID (in that order).
func (s *adminServer) queryDescriptorIDPath(
  ctx context.Context, session *sql.Session, names []string,

see above


pkg/server/admin.go, line 1403 at r1 (raw file):

  path := []sqlbase.ID{keys.RootNamespaceID}
  for _, name := range names {
      id, err := s.queryNamespaceID(ctx, session, path[len(path)-1], name)

see above


pkg/server/admin_test.go, line 243 at r1 (raw file):

  query := "CREATE DATABASE " + testdb
  createRes := ts.sqlExecutor.ExecuteStatements(session, query, nil)
  defer createRes.Close(ctx)

session.context


pkg/server/admin_test.go, line 271 at r1 (raw file):

  grantQuery := "GRANT " + strings.Join(privileges, ", ") + " ON DATABASE " + testdb + " TO " + testuser
  grantRes := s.(*TestServer).sqlExecutor.ExecuteStatements(session, grantQuery, nil)
  defer grantRes.Close(ctx)

see above


pkg/server/admin_test.go, line 302 at r1 (raw file):

  // Verify Descriptor ID.
  path, err := ts.admin.queryDescriptorIDPath(ctx, session, []string{testdb})

see above


pkg/server/admin_test.go, line 408 at r1 (raw file):

  for _, q := range setupQueries {
      res := ts.sqlExecutor.ExecuteStatements(session, q, nil)
      defer res.Close(ctx)

see above


pkg/server/admin_test.go, line 487 at r1 (raw file):

      resSet := ts.sqlExecutor.ExecuteStatements(session, showCreateTableQuery, nil)
      defer resSet.Close(ctx)

see above


pkg/server/admin_test.go, line 577 at r1 (raw file):

      resSet := ts.sqlExecutor.ExecuteStatements(session, showCreateTableQuery, nil)
      defer resSet.Close(ctx)

see above


pkg/server/admin_test.go, line 616 at r1 (raw file):

  for _, q := range setupQueries {
      res := ts.sqlExecutor.ExecuteStatements(session, q, nil)
      defer res.Close(ctx)

see above


pkg/server/admin_test.go, line 673 at r1 (raw file):

      params.SetValue(`2`, parser.NewDBytes(parser.DBytes(zoneBytes)))
      res := ts.sqlExecutor.ExecuteStatements(session, query, params)
      defer res.Close(ctx)

see above


pkg/server/admin_test.go, line 725 at r1 (raw file):

VALUES ('admin', 'abc'), ('bob', 'xyz')`
  res := ts.sqlExecutor.ExecuteStatements(session, query, nil)
  defer res.Close(ctx)

see above


pkg/server/admin_test.go, line 776 at r1 (raw file):

  for _, q := range setupQueries {
      res := ts.sqlExecutor.ExecuteStatements(session, q, nil)
      defer res.Close(ctx)

see above


pkg/sql/executor.go, line 577 at r1 (raw file):

          if results != nil {
              // Some results were produced by a previous attempt. Discard them.
              ResultList(results).Close(session.Ctx())

session.context. For sure.


pkg/sql/executor.go, line 1048 at r1 (raw file):

  if err != nil {
      if result.Rows != nil {
          result.Rows.Close(session.Ctx())

ditto


pkg/sql/executor.go, line 1224 at r1 (raw file):

              row = append(row, val)
          }
          if err := result.Rows.AddRow(planMaker.session.Ctx(), row); err != nil {

ditto


pkg/sql/explain.go, line 114 at r1 (raw file):

      }
      node := &explainTypesNode{
          ctx:      p.ctx(),

capture p not ctx


pkg/sql/explain.go, line 133 at r1 (raw file):

      node := &explainPlanNode{
          ctx:     p.ctx(),

ditto


pkg/sql/explain.go, line 182 at r1 (raw file):

func (e *explainTypesNode) Start() error {
  return populateTypes(e.ctx, e.results, e.plan, 0)

drop the argument, see below


pkg/sql/explain.go, line 210 at r1 (raw file):

}

func populateTypes(ctx context.Context, v *valuesNode, plan planNode, level int) error {

make this a method of *planner, use p.session.TxnState.Ctx


pkg/sql/explain.go, line 257 at r1 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

ditto

see above

pkg/sql/explain.go, line 286 at r1 (raw file):

func (e *explainPlanNode) Start() error {
  return populateExplain(e.ctx, e.verbose, e.results, e.plan, 0)

see above


pkg/sql/explain.go, line 295 at r1 (raw file):

func populateExplain(
  ctx context.Context, verbose bool, v *valuesNode, plan planNode, level int,

method of *planner, use p.session.TxnState.Ctx


pkg/sql/group.go, line 96 at r1 (raw file):

  group := &groupNode{
      planner: p,
      ctx:     p.ctx(),

Drop this


pkg/sql/group.go, line 97 at r1 (raw file):

      planner: p,
      ctx:     p.ctx(),
      values:  valuesNode{columns: s.columns, ctx: p.ctx()},

capture p not ctx


pkg/sql/group.go, line 174 at r1 (raw file):

type groupNode struct {
  planner *planner
  ctx     context.Context

drop this


pkg/sql/group.go, line 368 at r1 (raw file):

      }

      if err := n.values.rows.AddRow(n.ctx, row); err != nil {

p.session.TxnState.Ctx


pkg/sql/show.go, line 80 at r1 (raw file):

                  gen := varGen[vName]
                  value := gen(p)
                  if err := v.rows.AddRow(p.ctx(), parser.DTuple{parser.NewDString(vName),

p.session.TxnState.Ctx


pkg/sql/show.go, line 82 at r1 (raw file):

                  if err := v.rows.AddRow(p.ctx(), parser.DTuple{parser.NewDString(vName),
                      parser.NewDString(value)}); err != nil {
                      v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 144 at r1 (raw file):

                  defaultExpr,
              }
              if err := v.rows.AddRow(p.ctx(), newRow); err != nil {

ditto


pkg/sql/show.go, line 145 at r1 (raw file):

              }
              if err := v.rows.AddRow(p.ctx(), newRow); err != nil {
                  v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 236 at r1 (raw file):

              interleave, err := p.showCreateInterleave(&idx)
              if err != nil {
                  v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 272 at r1 (raw file):

          interleave, err := p.showCreateInterleave(&desc.PrimaryIndex)
          if err != nil {
              v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 277 at r1 (raw file):

          buf.WriteString(interleave)

          if err := v.rows.AddRow(p.ctx(), parser.DTuple{

ditto


pkg/sql/show.go, line 281 at r1 (raw file):

              parser.NewDString(buf.String()),
          }); err != nil {
              v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 367 at r1 (raw file):

          fmt.Fprintf(&buf, "AS %s", desc.ViewQuery)
          if err := v.rows.AddRow(p.ctx(), parser.DTuple{

ditto


pkg/sql/show.go, line 371 at r1 (raw file):

              parser.NewDString(buf.String()),
          }); err != nil {
              v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 402 at r1 (raw file):

          v := p.newContainerValuesNode(columns, 0)
          for _, db := range p.session.virtualSchemas.orderedNames {
              if err := v.rows.AddRow(p.ctx(), parser.DTuple{parser.NewDString(db)}); err != nil {

ditto


pkg/sql/show.go, line 403 at r1 (raw file):

          for _, db := range p.session.virtualSchemas.orderedNames {
              if err := v.rows.AddRow(p.ctx(), parser.DTuple{parser.NewDString(db)}); err != nil {
                  v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 411 at r1 (raw file):

                  bytes.TrimPrefix(row.Key, prefix), nil)
              if err != nil {
                  v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 414 at r1 (raw file):

                  return nil, err
              }
              if err := v.rows.AddRow(p.ctx(), parser.DTuple{parser.NewDString(name)}); err != nil {

ditto


pkg/sql/show.go, line 415 at r1 (raw file):

              }
              if err := v.rows.AddRow(p.ctx(), parser.DTuple{parser.NewDString(name)}); err != nil {
                  v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 476 at r1 (raw file):

                      parser.NewDString(userPriv.PrivilegeString()),
                  }
                  if err := v.rows.AddRow(p.ctx(), newRow); err != nil {

ditto


pkg/sql/show.go, line 477 at r1 (raw file):

                  }
                  if err := v.rows.AddRow(p.ctx(), newRow); err != nil {
                      v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 533 at r1 (raw file):

                  parser.MakeDBool(parser.DBool(isStored)),
              }
              return v.rows.AddRow(p.ctx(), newRow)

ditto


pkg/sql/show.go, line 540 at r1 (raw file):

              for i, col := range index.ColumnNames {
                  if err := appendRow(index, col, sequence, index.ColumnDirections[i].String(), false); err != nil {
                      v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 547 at r1 (raw file):

              for _, col := range index.StoreColumnNames {
                  if err := appendRow(index, col, sequence, "N/A", true); err != nil {
                      v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 615 at r1 (raw file):

                  detailsDatum,
              }
              if err := v.rows.AddRow(p.ctx(), newRow); err != nil {

ditto


pkg/sql/show.go, line 616 at r1 (raw file):

              }
              if err := v.rows.AddRow(p.ctx(), newRow); err != nil {
                  v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 682 at r1 (raw file):

              }

              if err := v.rows.AddRow(p.ctx(), parser.DTuple{parser.NewDString(tableName)}); err != nil {

ditto


pkg/sql/show.go, line 683 at r1 (raw file):

              if err := v.rows.AddRow(p.ctx(), parser.DTuple{parser.NewDString(tableName)}); err != nil {
                  v.rows.Close(p.ctx())

ditto


pkg/sql/show.go, line 733 at r1 (raw file):

                  parser.NewDString(f.Info),
              }
              if err := v.rows.AddRow(p.ctx(), row); err != nil {

ditto


pkg/sql/show.go, line 734 at r1 (raw file):

              }
              if err := v.rows.AddRow(p.ctx(), row); err != nil {
                  v.rows.Close(p.ctx())

ditto


pkg/sql/sort.go, line 324 at r1 (raw file):

          v.ordering = n.ordering
          n.sortStrategy = newSortAllStrategy(v)
          n.sortStrategy.Finish(n.ctx)

n.p.session.TxnState.Ctx


pkg/sql/sort.go, line 343 at r1 (raw file):

      }
      if !next {
          n.sortStrategy.Finish(n.ctx)

ditto


pkg/sql/table_join.go, line 44 at r1 (raw file):

  joinType joinType

  ctx context.Context

capture planner not ctx


pkg/sql/table_join.go, line 525 at r1 (raw file):

      info: info,
      plan: &joinNode{
          ctx:       p.ctx(),

see above


pkg/sql/table_join.go, line 631 at r1 (raw file):

          newRow := make([]parser.Datum, len(row))
          copy(newRow, row)
          if err := n.rightRows.rows.AddRow(n.ctx, newRow); err != nil {

n.p.session.TxnState.Ctx


pkg/sql/values.go, line 35 at r1 (raw file):

type valuesNode struct {
  n        *parser.ValuesClause
  ctx      context.Context

Remove this; only use the planner reference. See below


pkg/sql/values.go, line 52 at r1 (raw file):

func (p *planner) newContainerValuesNode(columns ResultColumns, capacity int) *valuesNode {
  return &valuesNode{
      ctx:     p.ctx(),

That's incorrect; the containerValuesNode should bind p not p.ctx(). See below.


pkg/sql/values.go, line 62 at r1 (raw file):

) (planNode, error) {
  v := &valuesNode{
      ctx:          p.ctx(),

Remove this.


pkg/sql/values.go, line 166 at r1 (raw file):

          }
      }
      if err := n.rows.AddRow(n.p.ctx(), row); err != nil {

n.p.session.TxnState.Ctx


pkg/sql/values.go, line 208 at r1 (raw file):

func (n *valuesNode) Close() {
  if n.rows != nil {
      n.rows.Close(n.ctx)

n.p.session.TxnState.Ctx


pkg/sql/values.go, line 258 at r1 (raw file):

// Push implements the heap.Interface interface.
func (n *valuesNode) Push(x interface{}) {
  n.err = n.rows.AddRow(n.ctx, n.tmpValues)

n.p.session.TxnState.Ctx


pkg/sql/virtual_schema.go, line 124 at r1 (raw file):

              }
          }
          return v.rows.AddRow(p.ctx(), datums)

p.session.TxnState.Ctx


pkg/sql/window.go, line 76 at r1 (raw file):

  window := &windowNode{
      planner:      p,
      values:       valuesNode{columns: s.columns, ctx: p.ctx()},

Remove this ctx init


pkg/sql/window.go, line 448 at r1 (raw file):

      windowDefEnd := renderEnd + n.wrappedWindowDefVals.NumCols()
      renderVals := valuesCopy[:renderEnd]
      if err := n.wrappedRenderVals.AddRow(n.planner.ctx(), renderVals); err != nil {

n.planner.session.TxnState.Ctx


pkg/sql/window.go, line 452 at r1 (raw file):

      }
      windowDefVals := valuesCopy[renderEnd:windowDefEnd]
      if err := n.wrappedWindowDefVals.AddRow(n.planner.ctx(), windowDefVals); err != nil {

n.planner.session.TxnState.Ctx


pkg/sql/window.go, line 456 at r1 (raw file):

      }
      indexedVarVals := valuesCopy[windowDefEnd:]
      if err := n.wrappedIndexedVarVals.AddRow(n.planner.ctx(), indexedVarVals); err != nil {

n.planner.session.TxnState.Ctx


pkg/sql/window.go, line 677 at r1 (raw file):

  // Done using window definition values, release memory.
  n.wrappedWindowDefVals.Close(n.planner.ctx())

n.planner.session.TxnState.Ctx


pkg/sql/window.go, line 737 at r1 (raw file):

      }

      if err := n.values.rows.AddRow(n.planner.ctx(), curRow); err != nil {

n.planner.session.TxnState.Ctx


pkg/sql/window.go, line 744 at r1 (raw file):

  // Done using the output of computeWindows, release memory and clear
  // accounts.
  n.wrappedRenderVals.Close(n.planner.ctx())

n.planner.session.TxnState.Ctx


pkg/sql/window.go, line 746 at r1 (raw file):

  n.wrappedRenderVals.Close(n.planner.ctx())
  n.wrappedRenderVals = nil
  n.wrappedIndexedVarVals.Close(n.planner.ctx())

n.planner.session.TxnState.Ctx


pkg/sql/window.go, line 786 at r1 (raw file):

  n.plan.Close()
  if n.wrappedRenderVals != nil {
      n.wrappedRenderVals.Close(n.planner.ctx())

n.planner.session.TxnState.Ctx


pkg/sql/window.go, line 790 at r1 (raw file):

  }
  if n.wrappedWindowDefVals != nil {
      n.wrappedWindowDefVals.Close(n.planner.ctx())

n.planner.session.TxnState.Ctx


pkg/sql/window.go, line 794 at r1 (raw file):

  }
  if n.wrappedIndexedVarVals != nil {
      n.wrappedIndexedVarVals.Close(n.planner.ctx())

n.planner.session.TxnState.Ctx


pkg/sql/pgwire/v3.go, line 747 at r1 (raw file):

  // has been bound by v3Conn.setupSession().
  results := c.executor.ExecuteStatements(c.session, stmts, pinfo)
  defer results.Close(ctx)

probably c.session.context here


Comments from Reviewable

@andreimatei
Copy link
Contributor Author

I think having the nodes figure out if they want the session ctx or the txn context would be going the wrong way. I think they never want anything other than the inner-most context available to them. And the question is whether that can be anything other than the transaction one... And I think the problem is with preparing... because that can happen with or without a transaction being active (right?).
Maybe we should pass a ctx to every method on a planNode... On the other hand, at least currently, I don't quite see a clear advantage versus the somewhat magic session.Ctx() method, given that the session/txn ctx can only change when no planNodes are in scope (at least once we get rid of EXPLAIN(TRACE)).


Review status: all files reviewed at latest revision, 102 unresolved discussions, some commit checks failed.


Comments from Reviewable

@andreimatei
Copy link
Contributor Author

I've replied to the comments on the first few files, up to where the planNodes start.


Review status: all files reviewed at latest revision, 102 unresolved discussions, some commit checks failed.


pkg/server/admin.go, line 190 at r1 (raw file):

Previously, knz (kena) wrote…

probably r.Close(session.context)

You're almost right that that's the context that should be used: we do want to use a context annotated by the server, but the session's context is also annotated by the executor. Even ignoring that, I think this function, and all the others in this file, should pass a ctxs explicitly and be oblivious to what the session is doing internally.

I've changed NewSessionForRPC to also return the annotated ctx. See if you like it.


pkg/server/admin.go, line 276 at r1 (raw file):

Previously, knz (kena) wrote…

not sure this is needed, you can pull it from session

mmm, I prefer passing it directly (and now it's the correct one)

pkg/server/admin_test.go, line 243 at r1 (raw file):

Previously, knz (kena) wrote…

session.context

nope, I think in this test file we're good; `ctx` has been properly annotated above.

pkg/sql/executor.go, line 577 at r1 (raw file):

Previously, knz (kena) wrote…

session.context. For sure.

no... The transaction's ctx should be used, on both the allocation side (below, where you've had the same comment), and the deallocation side, here. You always want to use the "current" context for all your operations. Also note that even if there was a question about whether the same context had or had not been used on the allocation side, it wouldn't matter; you'd still want to use the current context for deallocation. In fact, the whole point of this PR is to let structures like this `results` be deallocated in a different context than the one in which they were created (although that's not actually happening in this case). If you're somehow just tracing this transaction, for example, and this deallocation call takes 3 seconds and produces a bunch of log messages, you'd want to get those messages.

Right?


Comments from Reviewable

@knz
Copy link
Contributor

knz commented Nov 6, 2016

Unless you're in a hurry I'd like to spend a few days mulling over alternatives. Let's chat Friday.

@tamird
Copy link
Contributor

tamird commented Nov 6, 2016

Reviewed 1 of 2 files at r2.
Review status: 16 of 17 files reviewed at latest revision, 102 unresolved discussions, some commit checks failed.


Comments from Reviewable

@andreimatei
Copy link
Contributor Author

I've rewritten this PR, plumbing contexts through the planNode interface and in a bunch of places through the sql code. Please take another look :)

cc @tamird


Review status: 0 of 101 files reviewed at latest revision, 102 unresolved discussions, some commit checks failed.


Comments from Reviewable

@tamird
Copy link
Contributor

tamird commented Feb 27, 2017

Reviewed 3 of 83 files at r3.
Review status: 0 of 101 files reviewed at latest revision, 103 unresolved discussions, some commit checks failed.


pkg/sql/parser/eval.go, line 1744 at r3 (raw file):

	// from the Session.
	// !!! set this
	Ctx context.Context

why? is this really just to avoid renaming "ctx" in existing methods?


Comments from Reviewable

@tamird
Copy link
Contributor

tamird commented Feb 27, 2017

To say that this is hard to review would be an understatement. You've removed some stored contexts, added others, changed some plumbing, and all this is spread out over multiple commits that don't appear to be meaningfully distinct.

What's the big picture? Why does this stop short of plumbing contexts everywhere they are needed? Which stored contexts now remain?


Reviewed 80 of 83 files at r3, 25 of 25 files at r4, 17 of 44 files at r5.
Review status: 74 of 101 files reviewed at latest revision, 111 unresolved discussions, some commit checks failed.


pkg/sql/alter_table.go, line 473 at r3 (raw file):

}

func (n *alterTableNode) Next(_ context.Context) (bool, error) { return false, nil }

throughout: you don't need this underscore. they're only needed when some of the arguments are named.


pkg/sql/distsql_physical_planner.go, line 1398 at r4 (raw file):

type distSQLReceiver struct {
	ctx context.Context

proliferating more stored contexts is not the way I'd like to see this achieved.


pkg/sql/executor.go, line 1383 at r3 (raw file):

	tStart time.Time,
) (Result, error) {
	// !!! execStmt should take a ctx, and this metod should stop planMaker.ctx()

well why doesn't it?


pkg/sql/lease.go, line 1257 at r3 (raw file):

						if t := m.findTableState(table.ID, false /* create */); t != nil {
							if err := t.purgeOldLeases(
								context.TODO(), db, table.Dropped(), table.Version, m); err != nil {

while you're here, can you extract all these contexts into a local?


pkg/sql/lease_test.go, line 202 at r3 (raw file):

	const descID = keys.LeaseTableID
	ctx := context.TODO()

i mean if you're going to do this, you can just testingT.Context() instead.


pkg/sql/schema_changer.go, line 840 at r3 (raw file):

				for tableID, sc := range s.schemaChangers {
					if timeutil.Since(sc.execAfter) > 0 {
						// !!! create annotated ctx for schema changes.

?


pkg/sql/subquery.go, line 286 at r3 (raw file):

	err     error

	// TODO(andrei): plumb the context through the parser.Visitor.

again, why not finish this here?


pkg/sql/walk.go, line 75 at r3 (raw file):

}

// makePlanVisitor creates a planVisitor instance.

👎


pkg/sql/pgwire/v3.go, line 336 at r4 (raw file):

		return err
	}
	// Now that a Session has been set up, further operations done on behalf of

why not replace this comment with ctx = c.session.Ctx()?


Comments from Reviewable

@knz
Copy link
Contributor

knz commented Feb 28, 2017

Side note: this may fix #13837.

@knz
Copy link
Contributor

knz commented Feb 28, 2017

Reviewed 45 of 83 files at r3, 13 of 25 files at r4, 42 of 44 files at r5, 1 of 1 files at r6.
Review status: all files reviewed at latest revision, 43 unresolved discussions, some commit checks failed.


pkg/sql/distsql_physical_planner.go, line 1398 at r4 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

proliferating more stored contexts is not the way I'd like to see this achieved.

That's true, meanwhile I also think this can be addressed in a subsequent PR.


pkg/sql/executor.go, line 577 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

no... The transaction's ctx should be used, on both the allocation side (below, where you've had the same comment), and the deallocation side, here.
You always want to use the "current" context for all your operations.
Also note that even if there was a question about whether the same context had or had not been used on the allocation side, it wouldn't matter; you'd still want to use the current context for deallocation. In fact, the whole point of this PR is to let structures like this results be deallocated in a different context than the one in which they were created (although that's not actually happening in this case).
If you're somehow just tracing this transaction, for example, and this deallocation call takes 3 seconds and produces a bunch of log messages, you'd want to get those messages.

Right?

So I take it that we're keeping a context reference in the session after all?


pkg/sql/executor.go, line 1293 at r6 (raw file):

// implementation.
func (e *Executor) execClassic(planMaker *planner, plan planNode, result *Result) error {
	if err := planMaker.startPlan(planMaker.session.Ctx(), plan); err != nil {

Please store the context reference in a local variable here and reuse it throughout. Will increase legibility.


pkg/sql/lease.go, line 1257 at r3 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

while you're here, can you extract all these contexts into a local?

👍


pkg/sql/lease_test.go, line 202 at r3 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

i mean if you're going to do this, you can just testingT.Context() instead.

👍


pkg/sql/pg_catalog.go, line 1237 at r6 (raw file):

		// include sensitive information such as password hashes.
		h := makeOidHasher()
		return forEachUser(context.TODO(), p,

I'd extend the populate prototype to provide contexts instead of this.


pkg/sql/schema_changer.go, line 840 at r3 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

?

Please phrase the comment as a TODO and/or (preferred) file a follow-up issue.


pkg/sql/schema_changer.go, line 318 at r6 (raw file):

	defer func() {
		if err := sc.waitToUpdateLeases(ctx, sc.tableID); err != nil {
			log.Warning(context.TODO(), err)

log.Warning(ctx, err)


pkg/sql/schema_changer.go, line 337 at r6 (raw file):

	// errors that are resolved by retrying the backfill.
	if sqlbase.IsIntegrityConstraintError(err) {
		log.Warningf(context.TODO(), "reversing schema change due to irrecoverable error: %s", err)

log.Warning(ctx, ...)


pkg/sql/schema_changer.go, line 440 at r6 (raw file):

	}
	if log.V(2) {
		log.Infof(context.TODO(), "waiting for a single version of table %d...", tableID)

log.Infof(ctx, ... here and below.


pkg/sql/session.go, line 176 at r6 (raw file):

	// session abruptly in the middle of a transaction, or, until #7648 is
	// addressed, there might be leases accumulated by preparing statements.
	s.planner.releaseLeases(s.Ctx())

Why s.Ctx() here and not s.context?


pkg/sql/session.go, line 475 at r6 (raw file):

		panic("trying to execute schema changes while still in a transaction")
	}
	ctx := e.AnnotateCtx(context.TODO())

Is there no better context to plumb from the caller here?


pkg/sql/subquery.go, line 286 at r3 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

again, why not finish this here?

In this particular case I tend to disagree. I think that visitors whose work have a properly scoped lifetime should store the context. Stored context are not the devil, as long as the lifetime bounds are properly documented (please add a comment here to explain this)n


pkg/sql/trace.go, line 108 at r6 (raw file):

// sortNode.Next() after the explandPlanNode closes the tracing span (resulting
// in a span use-after-finish).
func (n *explainTraceNode) hijackTxnContext(ctx context.Context) error {

I would kinda prefer names other than "hijack" for these methods (here and on planner). Perhaps something based on "graft" instead? or "wrap"? or "instrument"?


pkg/sql/trace.go, line 146 at r6 (raw file):

	if first {
		n.rows = []parser.Datums{}
		if err := n.hijackTxnContext(ctx); err != nil {

Please clarify what happens if Close() is called just after Start() without any call to Next(). Is the call to n.unhijackCtx in Close() sound in this case?


pkg/sql/values_test.go, line 116 at r6 (raw file):

	for i, tc := range testCases {
		ctx := context.TODO()

t.Context? here and in other tests.


pkg/sql/walk.go, line 75 at r3 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

👎

YEah no. Binding the context in the visitor is fine but you don't need to create a new instance for that.


pkg/sql/distsqlrun/aggregator.go, line 120 at r6 (raw file):

		ag.outputTypes[i] = retType
	}
	if err := ag.out.init(post, ag.outputTypes, &flowCtx.evalCtx, output); err != nil {

?


pkg/sql/distsqlrun/aggregator_test.go, line 285 at r6 (raw file):

		flowCtx := FlowCtx{
			evalCtx: parser.EvalContext{},

?


pkg/sql/distsqlrun/distinct.go, line 50 at r6 (raw file):

	}

	if err := d.out.init(post, input.Types(), &flowCtx.evalCtx, output); err != nil {

?


pkg/sql/distsqlrun/flow.go, line 47 at r6 (raw file):

// FlowCtx encompasses the contexts needed for various flow components.
type FlowCtx struct {

I think this is time to document the fields of this struct.


pkg/sql/distsqlrun/flow.go, line 51 at r6 (raw file):

	id           FlowID
	evalCtx      parser.EvalContext

?


pkg/sql/distsqlrun/joinerbase.go, line 64 at r6 (raw file):

	types = append(types, rightTypes...)

	if err := jb.onCond.init(onExpr, types, &flowCtx.evalCtx); err != nil {

?


pkg/sql/distsqlrun/joinreader.go, line 73 at r6 (raw file):

	}

	if err := jr.out.init(post, types, &flowCtx.evalCtx, output); err != nil {

?


pkg/sql/distsqlrun/joinreader_test.go, line 99 at r6 (raw file):

	for _, c := range testCases {
		flowCtx := FlowCtx{
			evalCtx:  parser.EvalContext{},

?


pkg/sql/distsqlrun/mergejoiner_test.go, line 292 at r6 (raw file):

		rightInput := NewRowBuffer(nil, c.inputs[1])
		out := &RowBuffer{}
		flowCtx := FlowCtx{evalCtx: parser.EvalContext{}}

?


pkg/sql/distsqlrun/processors.go, line 231 at r6 (raw file):

) (*noopProcessor, error) {
	n := &noopProcessor{flowCtx: flowCtx, input: input}
	if err := n.out.init(post, input.Types(), &flowCtx.evalCtx, output); err != nil {

?


pkg/sql/distsqlrun/server.go, line 111 at r6 (raw file):

	ctx = flowCtx.AnnotateCtx(ctx)
	flowCtx.evalCtx.Ctx = func() context.Context {
		// TODO(andrei): This is wrong. Each processor should override Ctx with its

Can you extend this comment (or file an issue) to sketch how this should be done.


pkg/sql/distsqlrun/sorter.go, line 51 at r6 (raw file):

		limit:    spec.Limit,
	}
	if err := s.out.init(post, input.Types(), &flowCtx.evalCtx, output); err != nil {

?


pkg/sql/distsqlrun/tablereader.go, line 95 at r6 (raw file):

		types[i] = spec.Table.Columns[i].Type
	}
	if err := tr.out.init(post, types, &flowCtx.evalCtx, output); err != nil {

?


pkg/sql/distsqlrun/tablereader_test.go, line 118 at r6 (raw file):

		flowCtx := FlowCtx{
			evalCtx:  parser.EvalContext{},

?


pkg/sql/distsqlrun/values.go, line 51 at r6 (raw file):

		types[i] = v.columns[i].Type
	}
	if err := v.out.init(post, types, &flowCtx.evalCtx, output); err != nil {

?


pkg/sql/parser/eval.go, line 1720 at r6 (raw file):

// contextHolder is a wrapper that returns a Context.
type contextHolder func() context.Context

Why this type alias instead of type contextHolder *context.Context? Would be cheaper!


pkg/sql/parser/normalize.go, line 19 at r6 (raw file):

package parser

import (

I'm not sure this change belongs to this PR.


pkg/sql/pgwire/v3.go, line 336 at r4 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

why not replace this comment with ctx = c.session.Ctx()?

same question as tamir


Comments from Reviewable

@knz
Copy link
Contributor

knz commented Feb 28, 2017

Reviewed 1 of 83 files at r3, 1 of 44 files at r5.
Review status: all files reviewed at latest revision, 43 unresolved discussions, some commit checks failed.


Comments from Reviewable

@knz
Copy link
Contributor

knz commented Mar 1, 2017

(Discussed offline: the change in the flow interface is fine, but the rationale for it should be in the final commit message.)

@andreimatei
Copy link
Contributor Author

Review status: 30 of 102 files reviewed at latest revision, 43 unresolved discussions, some commit checks failed.


pkg/sql/alter_table.go, line 473 at r3 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

throughout: you don't need this underscore. they're only needed when some of the arguments are named.

Done.


pkg/sql/distsql_physical_planner.go, line 1398 at r4 (raw file):

Previously, knz (kena) wrote…

That's true, meanwhile I also think this can be addressed in a subsequent PR.

I'll leave this one for now... No easy solution


pkg/sql/executor.go, line 577 at r1 (raw file):

Previously, knz (kena) wrote…

So I take it that we're keeping a context reference in the session after all?

well you have to save a context somewhere... If a tracing span is to be used across queries, for example. I think keeping a ctx in the session makes sense.


pkg/sql/executor.go, line 1383 at r3 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

well why doesn't it?

Done.


pkg/sql/executor.go, line 1293 at r6 (raw file):

Previously, knz (kena) wrote…

Please store the context reference in a local variable here and reuse it throughout. Will increase legibility.

Done.


pkg/sql/group.go, line 174 at r1 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

ditto

Done.


pkg/sql/lease.go, line 1257 at r3 (raw file):

Previously, knz (kena) wrote…

👍

Done.


pkg/sql/lease_test.go, line 202 at r3 (raw file):

Previously, knz (kena) wrote…

👍

testingT.Context() is not a thing


pkg/sql/pg_catalog.go, line 1237 at r6 (raw file):

Previously, knz (kena) wrote…

I'd extend the populate prototype to provide contexts instead of this.

Done.


pkg/sql/schema_changer.go, line 840 at r3 (raw file):

Previously, knz (kena) wrote…

Please phrase the comment as a TODO and/or (preferred) file a follow-up issue.

added TODO


pkg/sql/schema_changer.go, line 318 at r6 (raw file):

Previously, knz (kena) wrote…

log.Warning(ctx, err)

Done.


pkg/sql/schema_changer.go, line 337 at r6 (raw file):

Previously, knz (kena) wrote…

log.Warning(ctx, ...)

Done.


pkg/sql/schema_changer.go, line 440 at r6 (raw file):

Previously, knz (kena) wrote…

log.Infof(ctx, ... here and below.

Done.


pkg/sql/session.go, line 176 at r6 (raw file):

Previously, knz (kena) wrote…

Why s.Ctx() here and not s.context?

changed


pkg/sql/session.go, line 475 at r6 (raw file):

Previously, knz (kena) wrote…

Is there no better context to plumb from the caller here?

there is indeed. done.


pkg/sql/subquery.go, line 286 at r3 (raw file):

Previously, knz (kena) wrote…

In this particular case I tend to disagree. I think that visitors whose work have a properly scoped lifetime should store the context. Stored context are not the devil, as long as the lifetime bounds are properly documented (please add a comment here to explain this)n

I could go either way on these visitors. They do have a limited lifespan, but also they have more than one external method (so you may want different contexts for the pre/post stages) and it somehow would feel natural for these methods to take a context. I'll leave this as is for now.


pkg/sql/trace.go, line 108 at r6 (raw file):

Previously, knz (kena) wrote…

I would kinda prefer names other than "hijack" for these methods (here and on planner). Perhaps something based on "graft" instead? or "wrap"? or "instrument"?

I think "hijack" signifies that something funky is going on - a layer that you wouldn't think should have any control over this session context wants to... hijack it.
I don't think there's any reason to rename in this PR. But once we get rid of the "explain trace" node and keep all the context manipulation in the executor or the sessions this particular method will go away and the other layers of "hijackTxnContext" can be renamed if they survive.


pkg/sql/trace.go, line 146 at r6 (raw file):

Previously, knz (kena) wrote…

Please clarify what happens if Close() is called just after Start() without any call to Next(). Is the call to n.unhijackCtx in Close() sound in this case?

the call to unhijackCtx in Close() is conditional...


pkg/sql/values.go, line 35 at r1 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

ditto

Done.


pkg/sql/values_test.go, line 116 at r6 (raw file):

Previously, knz (kena) wrote…

t.Context? here and in other tests.

not a thing


pkg/sql/walk.go, line 75 at r3 (raw file):

Previously, knz (kena) wrote…

YEah no. Binding the context in the visitor is fine but you don't need to create a new instance for that.

discussed with Rafa, there's nothing to do here. I'm not creating any more instances than before.
Tamir's -1 has been registered.


pkg/sql/distsqlrun/aggregator.go, line 120 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/distsqlrun/aggregator_test.go, line 285 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/distsqlrun/distinct.go, line 50 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/distsqlrun/flow.go, line 47 at r6 (raw file):

Previously, knz (kena) wrote…

I think this is time to document the fields of this struct.

the AmbientContext? I don't think we document it elsewhere... Not sure what to say other than the documentation on the type definition... I'll leave it alone.


pkg/sql/distsqlrun/flow.go, line 51 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/distsqlrun/joinerbase.go, line 64 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/distsqlrun/joinreader.go, line 73 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/distsqlrun/joinreader_test.go, line 99 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/distsqlrun/mergejoiner_test.go, line 292 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/distsqlrun/processors.go, line 231 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/distsqlrun/server.go, line 111 at r6 (raw file):

Previously, knz (kena) wrote…

Can you extend this comment (or file an issue) to sketch how this should be done.

I don't really have anything else to say at the moment :)
We should change how the evalCtx is initialized - we probably want each processor to initialize it instead of creating it here...


pkg/sql/distsqlrun/sorter.go, line 51 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/distsqlrun/tablereader.go, line 95 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/distsqlrun/tablereader_test.go, line 118 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/distsqlrun/values.go, line 51 at r6 (raw file):

Previously, knz (kena) wrote…

?

Done.


pkg/sql/mon/mem_usage.go, line 456 at r1 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

nit: you could remove this

Done.


pkg/sql/parser/eval.go, line 1744 at r3 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

why? is this really just to avoid renaming "ctx" in existing methods?

it's not about renaming, but about not changing the Eval interface. Thought some more after discussing with you; I again think there's no reason to change that interface and extract the ctx from the EvalCtx.
The type of this member (which becomes an indirect contextHolder func() context.Context in a later commit) should change, but that's a different issue.


pkg/sql/parser/eval.go, line 1720 at r6 (raw file):

Previously, knz (kena) wrote…

Why this type alias instead of type contextHolder *context.Context? Would be cheaper!

well because we'll be assigning this to session.Ctx(), which retrieves the context from either the session or the current transaction


pkg/sql/pgwire/v3.go, line 336 at r4 (raw file):

Previously, knz (kena) wrote…

same question as tamir

because session.Ctx() can evolve over time.


pkg/sql/parser/normalize.go, line 19 at r6 (raw file):

Previously, knz (kena) wrote…

I'm not sure this change belongs to this PR.

not intentional, removed.


Comments from Reviewable

@knz
Copy link
Contributor

knz commented Mar 7, 2017

Reviewed 18 of 73 files at r7, 13 of 24 files at r8, 42 of 43 files at r9, 1 of 1 files at r10.
Review status: all files reviewed at latest revision, 21 unresolved discussions, some commit checks failed.


pkg/sql/executor.go, line 874 at r5 (raw file):

	for i, stmt := range stmts {
		ctx := planMaker.session.Ctx()

Why did you remove this line? It seems to me that executing statements may change the txn context (e.g. via begin/commit), so we'd need to refresh the ctx reference at every iteration. Wouldn't we?


pkg/sql/lease.go, line 1217 at r10 (raw file):

func (m *LeaseManager) RefreshLeases(s *stop.Stopper, db *client.DB, gossip *gossip.Gossip) {
	s.RunWorker(func() {
		ctx := context.TODO()

Perhaps here either a TODO comment with a suggestion as to what needs to be done (ambient context? more plumbing?) and/or an issue to follow-up.


pkg/sql/trace.go, line 108 at r6 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

I think "hijack" signifies that something funky is going on - a layer that you wouldn't think should have any control over this session context wants to... hijack it.
I don't think there's any reason to rename in this PR. But once we get rid of the "explain trace" node and keep all the context manipulation in the executor or the sessions this particular method will go away and the other layers of "hijackTxnContext" can be renamed if they survive.

ack


pkg/sql/trace.go, line 146 at r6 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

the call to unhijackCtx in Close() is conditional...

But that's the thing: I do not see (without additional explanation) why this conditional is sufficient. What makes spanFinished change?


pkg/sql/distsqlrun/flow.go, line 47 at r6 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

the AmbientContext? I don't think we document it elsewhere... Not sure what to say other than the documentation on the type definition... I'll leave it alone.

I was thinking about documenting all the fields in this struct, not just this one.


pkg/sql/parser/eval.go, line 1720 at r6 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

well because we'll be assigning this to session.Ctx(), which retrieves the context from either the session or the current transaction

Oh ok carry on.


Comments from Reviewable

@andreimatei
Copy link
Contributor Author

Review status: all files reviewed at latest revision, 19 unresolved discussions, some commit checks failed.


pkg/sql/executor.go, line 874 at r5 (raw file):

Previously, knz (kena) wrote…

Why did you remove this line? It seems to me that executing statements may change the txn context (e.g. via begin/commit), so we'd need to refresh the ctx reference at every iteration. Wouldn't we?

you're right. As we've discussed, I've tried to plumb a *context.Context throughout the Executor, so that statements can update it, but didn't end up with something pretty. I've reverted the executor to always use session.Ctx.


pkg/sql/lease.go, line 1217 at r10 (raw file):

Previously, knz (kena) wrote…

Perhaps here either a TODO comment with a suggestion as to what needs to be done (ambient context? more plumbing?) and/or an issue to follow-up.

I don't have anything insightful to write in an issue, and there's still context.TODO() around the code... I think this is good enough.


pkg/sql/trace.go, line 146 at r6 (raw file):

Previously, knz (kena) wrote…

But that's the thing: I do not see (without additional explanation) why this conditional is sufficient. What makes spanFinished change?

the relevant condition was if n.tracingCtx == nil {return} in Close(). tracingCtx was only set if we actually hijacked.
But I think there was a bug there causing n.plan.Close() to not be called. In any case, I've simplified the code with a more clear condition. PTAL.


pkg/sql/distsqlrun/flow.go, line 47 at r6 (raw file):

Previously, knz (kena) wrote…

I was thinking about documenting all the fields in this struct, not just this one.

I've added a small commit commenting them. Did my best :P.


Comments from Reviewable

@knz
Copy link
Contributor

knz commented Mar 8, 2017

Reviewed 6 of 61 files at r11, 13 of 24 files at r12, 40 of 43 files at r13, 1 of 1 files at r14, 1 of 1 files at r15.
Review status: all files reviewed at latest revision, 20 unresolved discussions, some commit checks pending.


pkg/sql/lease.go, line 1217 at r10 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

I don't have anything insightful to write in an issue, and there's still context.TODO() around the code... I think this is good enough.

👍


pkg/sql/prepare.go, line 85 at r15 (raw file):

	// during prepare, this should be tallied up to the monitor as well.
	sz := int64(uintptr(len(query)+len(name)) + unsafe.Sizeof(*stmt))
	if err := stmt.memAcc.Wsession(ps.session).OpenAndInit(ps.session.Ctx(), sz); err != nil {

Well then I'd suggest making the methods of the WrappedAccount fish the context in their session attribute again instead of an argument. This improves readability.


pkg/sql/trace.go, line 146 at r6 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

the relevant condition was if n.tracingCtx == nil {return} in Close(). tracingCtx was only set if we actually hijacked.
But I think there was a bug there causing n.plan.Close() to not be called. In any case, I've simplified the code with a more clear condition. PTAL.

Nice, thanks.


pkg/sql/distsqlrun/flow.go, line 47 at r6 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

I've added a small commit commenting them. Did my best :P.

Excellent.


pkg/sql/parser/builtins.go, line 1571 at r15 (raw file):

			fn: func(ctx *EvalContext, args Datums) (Datum, error) {
				r, err := ctx.Planner.QueryRow(
					ctx.Ctx(), "SELECT indexdef FROM pg_catalog.pg_indexes WHERE crdb_oid=$1", args[0])

Doesn't seem that this belongs to this PR.


Comments from Reviewable

@andreimatei
Copy link
Contributor Author

Review status: 47 of 102 files reviewed at latest revision, 17 unresolved discussions, some commit checks failed.


pkg/sql/prepare.go, line 85 at r15 (raw file):

Previously, knz (kena) wrote…

Well then I'd suggest making the methods of the WrappedAccount fish the context in their session attribute again instead of an argument. This improves readability.

This sounds off. Ping me when you can :)


pkg/sql/parser/builtins.go, line 1571 at r15 (raw file):

Previously, knz (kena) wrote…

Doesn't seem that this belongs to this PR.

What doesn't belong? Depending on what diff you look at, you might see a oid -> args[0] change. But that's not my PR, it was done by someone else.


Comments from Reviewable

@tamird
Copy link
Contributor

tamird commented Mar 9, 2017

first commit message lies:

... and get rid of planner.Ctx() in a next commit.

that doesn't happen, afaict.

please consider squashing these commits:

sql: don't bind a Context in BoundAccount
sql: add a ctx to important planNode methods
sql, distsqlrun: get rid of planner.ctx()

the separation in this case does more harm than good. the TL;DR here is that storing a context is generally a bad thing.

other than that, :lgtm:


Reviewed 9 of 73 files at r7, 5 of 61 files at r11, 62 of 62 files at r16, 24 of 24 files at r17, 43 of 43 files at r18, 1 of 1 files at r19, 1 of 1 files at r20.
Review status: all files reviewed at latest revision, 10 unresolved discussions, all commit checks successful.


pkg/sql/planner.go, line 179 at r17 (raw file):

// ctx returns the current session context (suitable for logging/tracing).
func (p *planner) ctx() context.Context {

🎊 🎉


pkg/sql/sort.go, line 477 at r17 (raw file):

}

func (ss *sortAllStrategy) Finish(_ context.Context) {

several unnecessary underscores here (resolved in a later commit). seriously, please consider squashing!


pkg/sql/trace.go, line 99 at r18 (raw file):
when? ah, a later commit.

However, the commit message simply reads:

sql: change explanation in explainTraceNode

and this change is doing a bit more than that.


pkg/sql/values.go, line 34 at r17 (raw file):

type valuesNode struct {
	n        *parser.ValuesClause
	ctx      context.Context

this needs a comment explaining why it is done (the heap interface, i guess?)

heh, this was also resolved in a later commit. another plea for a squash.


pkg/sql/values.go, line 234 at r20 (raw file):

// Push implements the heap.Interface interface.
func (n *valuesNode) Push(x interface{}) {

while you're here you can s/x //


pkg/sql/distsqlrun/flow.go, line 50 at r20 (raw file):

	log.AmbientContext

	// id is a unique identified for a flow.

identifier


pkg/sql/distsqlrun/flow.go, line 52 at r20 (raw file):

	// id is a unique identified for a flow.
	id FlowID
	// evalCtx is used by all the processors in the flow to evaluate.

to evaluate what?


pkg/sql/distsqlrun/server.go, line 102 at r18 (raw file):

		AmbientContext: ds.AmbientContext,
		id:             req.Flow.FlowID,
		// TODO(andrei): more fields from evalCtx need to be initialized.

they do? which ones?


Comments from Reviewable

@andreimatei
Copy link
Contributor Author

squashed

Thanks for the review!


Review status: 98 of 102 files reviewed at latest revision, 9 unresolved discussions, all commit checks successful.


pkg/sql/prepare.go, line 85 at r15 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

This sounds off. Ping me when you can :)

discussed this at length and opinions unfortunately still differ. I'll merge like this, pending further consideration.

I was looking at some code, though. Seems to me that

  • a) WrappableMemoryAccount was always a no-op. And, in fact, the design is funky since {Session,txnState}.OpenAccount() are no-ops, but lead the caller to believe that they have something to do with the session/txnState objects they're called on. But the the resulting WrappableMemoryAccount can be bound to either the session or the txn monitor, regardless of how it was opened.
  • b) since I've made WrappedMemory not capture a ctx, it's currently identical to BoundAccount.

How about we get rid of both WrappableMemAcc and WrappedMemAcc, in favor of BoundAcc everywhere, and this way we avoid the discussion about whether the WrappedMemAcc should capture the ctx and have some special lifetime semantics? :P


pkg/sql/distsqlrun/flow.go, line 50 at r20 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

identifier

Done.


pkg/sql/distsqlrun/flow.go, line 52 at r20 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

to evaluate what?

Done.


pkg/sql/distsqlrun/server.go, line 102 at r18 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

they do? which ones?

referenced an issue


Comments from Reviewable

The objective is to be more explicit about context passing: don't store
the context in some planNodes and get rid of planner.Ctx().

- a ctx is added to important planNode methods
- don't bind a Context in BoundAccount*

references cockroachdb#7992

*a BoundAccount is a MemoryAccount bound to a MemoryMonitor. When
creating a BoundAcc, we used to bind a context to be used by the Acc
throughout its lifetime. This means that the context had to stay "valid"
throughout (e.g. the span in the ctx, if any, had to stay open). This is
conceptually a problem because it means you can't open an acc in one
place and close it in a completely different place (and also diminishes
the utility of contexts, as Account operations were not logged in a
context corresponding to the function actually performing them). In
particular, this was a problem for accounts created for RowContainers
and the way how EXPLAIN(TRACE) temporarily hijacks the session's
context: a container's account could easily be bound to one of these
temporarily hijacked contexts, and closed after the corresponding span
had already been Finish()ed.
This patch plumbs through a Context to the accounts, through various
cascading layers.
@tamird
Copy link
Contributor

tamird commented Mar 9, 2017

Reviewed 1 of 73 files at r7, 1 of 43 files at r18, 8 of 10 files at r21, 1 of 1 files at r22, 1 of 1 files at r23.
Review status: all files reviewed at latest revision, 6 unresolved discussions, some commit checks pending.


Comments from Reviewable

@andreimatei andreimatei merged commit 5761762 into cockroachdb:master Mar 9, 2017
@andreimatei andreimatei deleted the BoundAccount-ctx branch March 9, 2017 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants