METABUG: Tier 2 optimizer

[mdboom]

Note
This issue is not for discussion, but to keep the various steps organized. To create new subtasks, create new issues and link to them from here by editing this comment.

To discuss the Tier 2 optimizer topic in general, see #557.

Components

• VM hooks and callbacks -- Enable VM to tell optimizer about hotspots and to warn it of changes.
• Creation of superblocks #558
• Superblock optimization: Specialization #560
• Superblock optimization: Partial evaluation #561
• Micro-op (tier 2) interpreter #580
• Deoptimization of superblocks #563

Design artifacts

• Optimizer API
• Optimizer instruction format
• Tier 2 interpreter instruction format (if different from the optimizer instruction format)
• Executable object layout

Dependencies

graph TD;
  optimizer_api[Optimizer API] --> superblock_generator[Superblock generator];
  optimizer_format[Optimizer instruction format] --> superblock_generator;
  superblock_generator --> specializer[Specializer];
  superblock_generator --> partial_evaluator[Partial evaluator];
  superblock_generator --> compiler[Compiler];
  superblock_generator --> deoptimization[Deoptimization of superblocks];
  deoptimization[Deoptimization of superblocks] --> partial_evaluator[Partial evaluator];
  optimizer_format --> interpreter_format[Interpreter instruction format];
  interpreter_format --> interpreter[Interpreter];
  executable_object_layout[Executable object layout] --> interpreter;
  executable_object_layout[Executable object layout] --> compiler;
  interpreter --> specializer;
  interpreter --> partial_evaluator;
  interpreter --> compiler;

[markshannon]

Although the development of the partial evaluator and compiler can proceed in parallel with that of the specializer, it is worth noting that the partial evaluator and compiler have little value without the specializer. The specializer must take priority over the other optimizer components.