-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
0 parents
commit e78ff77
Showing
61 changed files
with
6,365 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Sphinx build info version 1 | ||
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. | ||
config: 7b627cd2879137a7bc75fc959d5d7381 | ||
tags: 645f666f9bcd5a90fca523b33c5a78b7 |
Empty file.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
Features | ||
======== | ||
|
||
GHC Status | ||
---------- | ||
|
||
- GHC Process information | ||
|
||
.. image:: images/screenshots/ghc-process-info.png | ||
:width: 480 | ||
|
||
- Current modules in compilation | ||
|
||
.. image:: images/screenshots/current-modules-being-compiled.png | ||
:width: 320 | ||
|
||
Build Information and Performance | ||
--------------------------------- | ||
|
||
- Build Graph | ||
|
||
.. image:: images/screenshots/build-graph.png | ||
:width: 480 | ||
|
||
- Build Timing and Memory | ||
|
||
.. image:: images/screenshots/build-timing-memory.png | ||
:width: 480 | ||
|
||
- Blocker module information | ||
|
||
.. image:: images/screenshots/build-graph-blocker.png | ||
:width: 320 | ||
|
||
.. image:: images/screenshots/blocker-modules.png | ||
:width: 480 | ||
|
||
Interactive inspection | ||
---------------------- | ||
|
||
- Pause/Resume/Breakpoint/Console | ||
|
||
.. image:: images/screenshots/console-on-pause.png | ||
:width: 480 | ||
|
||
.. image:: images/screenshots/show-ghc-core.png | ||
:width: 480 | ||
|
||
- Source View | ||
|
||
Features under development | ||
-------------------------- | ||
|
||
- ghc-debug integration | ||
- ghci integration |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
Getting Started | ||
--------------- | ||
|
||
- Configuration | ||
|
||
One can configure ghc-specter by the file ``ghc-specter.yaml``. | ||
The file should be present in the directory where ghc-specter-daemon | ||
runs and where a target ``ghc`` runs (usually the root directory of a | ||
cabal project). | ||
A sample config YAML file can be found in ``ghc-specter.yaml.sample``. | ||
|
||
- Run daemon | ||
|
||
.. code-block:: text | ||
$ nix develop | ||
$ cabal run ghc-specter-daemon:ghc-specter-daemon -- online | ||
- Run GHC in a project | ||
|
||
Run GHC with ``-fplugin Plugin.GHCSpecter`` and ``-plugin-package ghc-specter-plugin``. | ||
If ``ghc-specter-plugin`` is already specified as the dependency of the project, the latter is not necessary. | ||
|
||
Example: | ||
|
||
.. code-block:: text | ||
$ cabal build --ghc-options "-fplugin Plugin.GHCSpecter -plugin-package ghc-specter-plugin" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,159 @@ | ||
How ``ghc-specter`` works | ||
========================= | ||
|
||
As a plugin system that installs multiple plugins and hooks dynamically, | ||
``ghc-specter`` is very closely tied to the GHC internal implementation. | ||
In this section, we will briefly overview the GHC compiler system and | ||
exhibit how the ``ghc-specter-plugin`` works inside GHC and also show how | ||
the ``ghc-specter-plugin`` and ``ghc-specter-daemon`` interplay. | ||
|
||
Background on the GHC Pipeline | ||
------------------------------ | ||
|
||
GHC operates in multiple different modes and each invocation of the GHC process consists of | ||
a sequence of operations called phases in the pipeline. Roughly speaking, what each | ||
phase/sub-phase does is to transform an intermediate representation of a given program to | ||
another lower-level intermediate representation until it is finally transformed to an assembly | ||
or similar (for example, LLVM-IR) so that it can be consumed by external standard tools to create | ||
an executable binary object. The process varies in different modes undoubtedly. | ||
|
||
The compilation also depends on the configuration environment many of which are set up via | ||
CLI flags. Over the course, the pipeline enriches (and simplifies if unnecessary) its internal | ||
state which consists of unique identifiers, names and type checking information. | ||
All of this persistent or transient information are of the interest of ``ghc-specter`` and we | ||
will try to capture and dump the information as requested. To hijack the pipeline, we rely on | ||
the GHC pipeline via the GHC plugin mechanism to install such inspection procedures. | ||
|
||
In the following, we expand the above description in detail. | ||
Note that The information here is presented for the sake of overview and | ||
mainly based on the GHC 9.4 implementation, and is neither | ||
precise nor comprehensive. In addition, for GHC 9.2, there are significant | ||
differences -- in particular, the GHC phases, -- so one should refer to the | ||
GHC documentation and the source code for more accurate information. | ||
|
||
|
||
GHC Mode | ||
^^^^^^^^ | ||
|
||
When invoked from the Command-Line-Interface (CLI), GHC is run in a certain mode. | ||
As we all are familiar, GHC can be used as both compiler and interpreter. | ||
In fact, this is one of the special luxurious features of GHC that we Haskell | ||
developers all enjoy. GHC compiler seamlessly integrates the interpreter | ||
read-eval-print-loop (REPL) session. GHC interpreter and the Run-Time-System (RTS) | ||
allows one to mix interpreted codes with other precompiled native binary | ||
bytecode. | ||
|
||
The compiler mode (or batch mode) can be run (i) in compilation manager mode | ||
(``--make``) in which a single GHC session will make a build plan (by topolically | ||
ordering module graph) and compile a group of modules associated with their source | ||
codes, or (ii) in oneshot mode (``-c``) in which only a single module is built for | ||
that GHC process. As the compilation starts, per module, a single GHC pipeline | ||
is carried out in a sequence of phases as explained in the next subsection. | ||
|
||
The interpreter mode compiles source codes or load precompiled modules dynamically | ||
from the user demand in a REPL session. Therefore, the interpreter mode is a | ||
trampoline process between a REPL session and compilation pipeline. When compiling | ||
a source file as demanded, the same GHC pipeline is invoked, so the plugins | ||
associated to each phase will be invoked as the compiler mode. But the backend | ||
phases for generating Cmm and assembly codes and linking is not performed since | ||
it is unnecessary. | ||
|
||
Phases and internal representation | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
A single module is being compiled in a single pipeline thread. In large chunks, | ||
the pipeline consists of the frontend phases, the backend phases and the external | ||
assembler/linker phases. The frontend phases starts with source codes, and | ||
parsing, typechecking, desugaring, and results in GHC Core. The GHC Core is | ||
optimized in a few Core-to-Core passes. Then, the backend phases generate | ||
the STG and Cmm representation, and finally generates assembly codes or LLVM IR | ||
depending on the configuration. Afterwards, the external assembler and linker | ||
(such as gcc or clang toolchain) will create a linkable binary. | ||
|
||
The frontend phase may start with a preprocessor (such as ``hsc2hs``). The | ||
compilation may end before the external linking process. For example, for the | ||
GHC interpreter mode, the compilation ends with producing bytecode binary and then | ||
the interpreter RTS will load and execute the bytecode directly. | ||
|
||
.. image:: images/GHC_Pipeline.png | ||
:align: left | ||
:width: 320 | ||
|
||
We list the relevant phases as defined in the GHC API 9.4 in the following: | ||
|
||
- ``Cpp``, ``HsPp``: C and Haskell preprocessor | ||
- ``Hsc``: Haskell frontend compiler up to type-checker | ||
- ``HscPostTc``: Haskell frontend compiler after type-checker | ||
- ``HscBackend``: Core backend compiler | ||
- ``Cmm``: C-- compiler | ||
- ``As``: External assembler | ||
- ``LlvmOpt``, ``LlvmLlc``, ``LlvmMangle``: LLVM | ||
- ``MergeForeign``: linker for foreign source | ||
|
||
In each phase, there can be smaller steps, which is called a `pass` in the GHC | ||
terminology. | ||
|
||
In the above, we briefly mentioned that each compilation phase produces a certain | ||
intermediate representation of the Haskell program. In a sense, the compiler is | ||
a sequence of transformation steps of such representations: | ||
|
||
- Source code and preprocessed code stored in file text file | ||
- ``HsParsedModule`` containing High-level Haskell declaration AST ``HsDecl`` | ||
(containing ``HsBind`` and ``HsExpr``) annotated with | ||
``Hsc``-phase-specific ``pass``-dependent information. | ||
Here, ``pass`` is progressed from ``Parsed`` through ``Renamed`` to ``Typechecked``. | ||
- GHC-Core-level ``CoreProgram`` which consists of a list of ``CoreBind`` (containing ``CoreExpr``) | ||
- STG-level ``StgTopBinding`` (``StgBinidng`` and ``StgExpr``) | ||
- Cmm-level ``CmmDecl`` | ||
- Generated assembly or LLVM bitcode in generated files | ||
|
||
|
||
|
||
Environment and state | ||
^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
Various configurations are determined before kicking off the compilation process. The CLI parameters | ||
are parsed and adjusted (by default or determined by system environment) with a few dynamically linked | ||
objects -- GHC plugins are such objects. The resultant configuration is stored in ``DynFlags`` data type, | ||
so making the content of ``DynFlags`` available to users is very useful. | ||
|
||
GHC is invoked in the ``CompManager`` mode, then the module dependency graph is analyzed and topologically | ||
sorted. Then, each module compilation is swept over the ordered list with the global shared environment | ||
``HscEnv``. The module graph information is stored in ``ModGraph``. | ||
|
||
Right after parsing the source code, the names in the parsed module are extracted but not yet bound to known | ||
imported/declared names, and conflicting names are not resolved with unique IDs. Therefore, the compiler | ||
does the ``Renamed`` pass. The information extracted and resolved should be accumulated into a global map, | ||
``GlobalRdrEnv``. After the name resolution and issuing unique IDs, typechecker pass will follow and develop | ||
another global map as a part of compilation state, collecting typechecking evidences. | ||
Combining such maps with the fully resolved Haskell AST, the accumulated ``TcGblEnv`` is produced as the | ||
result of the frontend typechecker and passed to the ``HscPostTc`` phase. | ||
|
||
|
||
Plugins and Hooks | ||
^^^^^^^^^^^^^^^^^ | ||
|
||
- GHC plugin, by which one can insert a custom code at a predefined plugin location in the compilation | ||
- GHC hooks, with which, one can replace a particular segment of compilation by a custom code. | ||
|
||
``ghc-specter`` System Design | ||
----------------------------- | ||
|
||
.. image:: images/ghc-specter-architecture.png | ||
:align: left | ||
:width: 480 | ||
|
||
The left figure shows the configuration of `ghc-specter` application. The GHC plugin | ||
`ghc-specter-plugin` is instantiated by invoking GHC with the CLI argument | ||
`-fplugin Plugin.GHCSpecter`. The daemon process `ghc-specter-daemon` should | ||
be running as a separate process before the plugin instantiation. | ||
As designated by a configuration file (``ghc-specter.yaml`` by default), the | ||
communication channel through a Unix socket between GHC and the daemon is | ||
established. | ||
|
||
We use ``driverPlugin`` as the starting point, and the plugin will install plugins and hooks dynamically. | ||
As plugins, ``parsedResultActionPlugin``, ``renamedResultAction``, ``spliceRunAction``, ``tcPlugin``, ``typecheckPlugin``, ``corePlugin`` that installs CoreToDo actions | ||
interleaved with existing CoreToDo steps. | ||
As hooks, we use ``runPhaseHook``, ``runMetaHook``, ``runPhaseHook``. | ||
|
||
The daemon is a web server to present the contents from `ghc-specter-plugin`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
``ghc-specter`` User's guide | ||
============================ | ||
|
||
.. toctree:: | ||
:maxdepth: 2 | ||
:caption: Contents: | ||
|
||
what-is-it | ||
how-it-works | ||
getting-started | ||
features | ||
|
||
Indices and tables | ||
================== | ||
|
||
* :ref:`genindex` | ||
* :ref:`modindex` | ||
* :ref:`search` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
What is ``ghc-specter``? | ||
======================== | ||
|
||
``ghc-specter`` is an inspecting tool and visualizer of internal | ||
information in the GHC compilation pipeline via GHC plugins and hooks. | ||
|
||
Motivation | ||
---------- | ||
|
||
Glasgow Haskell Compiler (GHC) is a de facto standard Haskell compiler. | ||
Haskell is special among popular programming languages | ||
as a pure Functional Programming (FP) language with lazy evaluation semantics | ||
and an advanced type system. | ||
The GHC compiler is a culmination of compiler research of decades to aggressively | ||
optimize lazily evaluated FP to the industry strength. Therefore, the compilation | ||
pipeline consists of a sequence of long and complex processes: | ||
parsing source, typechecking, transformation of Abstract Syntax Tree (AST) to | ||
simplified representation called GHC Core, optimization, and compilation into | ||
low-level and machine-level representation (STG, Cmm, Asm) and external assembler | ||
and linking steps against many architectures. | ||
|
||
Developers often need investigation in the middle of the pipeline, but have to | ||
rely only on built-in GHC logging or resort to a custom modification of the | ||
GHC source code. For example, we often need to identify blocking compilation steps | ||
dominating the build time, or to investigate a problematic Template Haskell (TH) | ||
code that is eating up all of the available memory. It is desirable to collect | ||
the relevant information right away from the very process of the GHC compilation | ||
and also intervene the process to inspect the internal state of the compiler. | ||
|
||
Fortunately, GHC provides excellent customization mechanisms for power users | ||
-- GHC plugins and GHC hooks, -- which do not require to rebuild the whole compiler and use | ||
that custom compiler for a project's build system. | ||
By simple command-line interface (CLI) flags, one can insert | ||
a custom code at a predefined plugin/hook location in the GHC compiler. | ||
|
||
|
||
ghc-specter consists of a GHC plugin (``ghc-specter-plugin``) and a web server process | ||
(``ghc-specter-daemon``). | ||
``ghc-specter-plugin`` installs a bunch of plugins and hooks into the build of a given | ||
project under inspection, and enables us to dump GHC's internal state in various | ||
available places. The information is being sent to ``ghc-specter-daemon``, which presents | ||
it visually for the sake of developer's convenience. As arbitrary custom code can be plugged | ||
in by a plugin, ghc-specter also provides interactive inspection method via a web console | ||
interface. | ||
|
||
Note that generically, one of the most lacking area of the Haskell ecosystem | ||
is to have good tooling for making internal idiosynchratic information available | ||
to users. ``ghc-specter`` is invented to fill the gap of the need. | ||
|
||
Comparison with HLS | ||
------------------- | ||
|
||
Haskell Language Server (HLS) is a Haskell IDE server daemon that | ||
implements Langauge Server Protocol (LSP) to improve Haskell developer experience | ||
with modern source code editor capabilities. There is a high-level overlap of the | ||
goal between HLS and ``ghc-specter``. However, they are very complementary to each other. | ||
|
||
HLS is a tool that helps Haskell programming tasks, and thus targets the editor | ||
integration and information indexing in order to shortening typical development | ||
iteration cycles. Therefore, its focus is leaning towards the frontend side of | ||
the GHC pipeline: parsing and typechecking. The daemon is designed to collect | ||
information over a longer span of development, not confining to a single run of | ||
GHC. Above all, it is not intended to collect the information from the actual | ||
build process. | ||
|
||
On the contrary, ghc-specter is to gather internal information even in a single | ||
invocation of GHC, and to render it available in an user-friendly form. | ||
As GHC is a special compiler with idiosyncratic internal information, we need | ||
a specialized presentation implementation, so we want graphical web interface | ||
and web console. The focus of the tool spans over all of the phases of the GHC | ||
pipeline, not only interested in the user-facing GHC frontend. | ||
It aims at easing identification of GHC-related problems and monitoring the | ||
process. | ||
|
||
Supported GHC | ||
------------- | ||
|
||
GHC 9.2 and 9.4 are supported as of the version 1.0.0.0. | ||
Because ghc-specter heavily relies on the GHC API and the API is changing | ||
wildly version by version, we will try to support 3 major versions of GHC or less. |
Oops, something went wrong.