From 63de79cd3dbbe98bc7a9ca678adf48eb130c5fef Mon Sep 17 00:00:00 2001 From: "Salve J. Nilsen" Date: Tue, 28 Aug 2012 23:48:43 +0200 Subject: [PATCH 1/2] Updated architecture docs to current state - Updated SVG, describing the compilation pipeline, with help from pmichaud++ - Updated the HTML prose describing the different steps in the pipeline, with help from pmichaud++ --- docs/architecture.html | 152 ++++---- docs/architecture.svg | 770 ++++++++++++++++++++--------------------- 2 files changed, 448 insertions(+), 474 deletions(-) diff --git a/docs/architecture.html b/docs/architecture.html index 08a04e312d1..a6f0f2f2a99 100644 --- a/docs/architecture.html +++ b/docs/architecture.html @@ -6,99 +6,89 @@

Rakudo architectural overview

-

(This started off as a blog -post, which didn't have the fancy clickable SVG image).

- + -

Parser and action methods

- -

The source code is entered at the top of the machine named Rakudo, and is -transformed in various stages. The first two, parser and action methods, are -actually part of Rakudo, and are hosted in the Rakudo repository. They are -written in two different subsets of Perl 6, the regexes (parser), and "Not -Quite Perl 6", short NQP (action methods).

- - -

PAST and POST -compiler

- -

The next two stages (PAST and POST compiler) are part of the so-called -"Parrot Compiler Toolkit", short PCT. Both PAST and POST are structural -representations of the program, with PAST being more high-level than POST. -Both compilers are written in PIR, the parrot -assembly language, and are distributed along with parrot. They are also used -by many other parrot based languages.

- - -

IMCC and Parrot runcores

- -

The POST compiler emits PIR, which IMCC transforms into byte code. IMCC is -parrot's PIR compiler, written in C and statically linked into parrot. The -byte code (PBC) can then be stored to disk, or executed in memory by a -so-called run core or run loop, which is in some sense -the heart -of parrot - or one of the hearts, because there are several different ones -available (one for just-in-time compilation (JIT), one for debugging etc.).

- -

There are also some supporting custom types and operations in Rakudo called -dynamic PMCs and dynamic ops which are written in C, and -helper functions written in other languages (namely NQP and PIR). Those do -not show up in the flow chart.

- -

Setting library

- -

The part of Rakudo described so far is the stage one compiler. -In the build process it is compiled first, and then it compiles the setting -library down to PBC. "Setting library" is a fancy term describing the -built-in functions which are written in Perl 6. The result of this compilation -(together with a low level runtime library in PIR) is linked together with the -stage one compiler and parrot, the result is the perl6 -executable.

+

Parser and Action methods

+ +

The Perl 6 source code is transformed in various stages. The first are the Parser and Action Method stage. This stage creates a parse tree out of the Perl 6 source code and the fires off Action methods that annotate the parse tree, incrementally turning it into an Abstract Syntax Tree (QAST). When an Action Method is done annotating, the control is handed back to the Parser, which then continues parsing the Perl 6 code.

+ +

The result of these two stages is an "improved PAST" (Perl 6 Abstract Syntax Tree) named QAST. This tree is then passed on to the QAST compiler.

+ +

The Parser and Action methods are implemented in "Not Quite Perl 6" (NQP) and are part of Rakudo and hosted in the Rakudo repository at src/Perl6/Grammar.pm and src/Perl6/Actions.pm.

+ +

QAST compiler

+ +

The QAST compiler transforms the abstract syntax tree into a PIRT (Parrot Intermediate Representation Tree). In this phase we have the program represented as an AST, and need to translate this into the PIR that the Parrot VM understands. To do this, the QAST compiler does a series of translations on the AST, creating PIRT nodes that implement the operations specified by the QAST nodes.

+ +

In addition, the QAST compiler is responsible for serializing The World in such a way that the later stages can get access to the declarations stored there during the Parser and Action methods stages.

+ +

There's also opportunity to apply some VM-specific optimizations at this point. When this is done, the resulting PIRT is passed to the PIRT serializer.

+ +

This stage is described in nqp/src/QAST/.

+ +

The World

+ +

The World is where the parser and the action methods store any declarations they encouter during their runs.

+ + +

PIRT serializer

+ +

The PIRT serializer "squashes" the PIR tree into a format that can be passed to Parrot itself and it's IMCC (InterMediate Code Compiler) stage.

+ +

You can read more about this at nqp/src/QAST/PIRT.nqp

+ + + +

IMCC and Parrot runtime

+ +

The IMCC (InterMediate Code Compiler) receives the PIR code from the PIRT serializer and then transforms it into Parrot Byte Code (PBC). IMCC is parrot's PIR compiler, written in C and statically linked into parrot. The byte code can then be stored to disk or executed in memory by one of the run cores availabe as part of the Parrot runtime. This is in some sense the heart of Parrot - or one of the hearts; There are several different cores available, includin one for just-in-time compilation (JIT), one for debugging and more.

+ +

You can find out more about this on parrot/docs/imcc/

+ +

PMCs and dynops

+ +

There are also some supporting custom types and operations in Rakudo called dynamic PMCs and dynamic ops (dynops) which are written in C, and helper functions written in NQP and PIR. These supporting libraries exist for adding features to Parrot that are needed to handle special features in Perl 6.

+ +

Core setting library

+ +

The part of Rakudo described so far is the stage one compiler. In the build process it is compiled first, and then it compiles the Core setting library down to PBC. "Core setting library" is a fancy term describing the built-in functions which are written in Perl 6, whose purpose is to be the Perl 6 standard library. The result of this compilation is linked together with the stage one compiler and Parrot, the result is the perl6 executable.

+ +

The core setting library informs the Parser and Action methods when compiling a Perl 6 program.

Glossary

-
PGE
-
Parrot Grammar Engine, parrot's grammar engine for Perl 6 regexes and - grammars.
-
NQP
-
Not Quite Perl 6, a small subset of Perl 6 that is used for tree - transformations in compilers.
- -
PAST
-
Parrot Abstract Syntax Tree, an in-memory representation of - structures common to many programming languages (like variable - declarations, branches, loops, subroutine calls).
- -
POST
-
Parrot Opcode Syntax Tree, an in-memory low level representation - of programs.
- -
PCT
-
Parrot Compiler Toolkit, a collection of tools and compilers useful - for writing other compilers.
- -
PIR
-
Parrot Intermediate Representation, the most commonly used for of - parrot assembly (which is still high-level enough to be written by - humans).
- -
IMCC
-
InterMediate Code Compiler, the part of parrot that compiles PIR - into byte code.
- -
PBC
-
Parrot Byte Code, the binary form to which all parrot programs are - compiled in the end.
+
Not Quite Perl 6, a small subset of Perl 6 that is used for tree transformations in compilers.
+ +
PIR
+
Parrot Intermediate Representation, the most commonly used for of parrot assembly (which is still high-level enough to be written by humans).
+ +
IMCC
+
InterMediate Code Compiler, the part of parrot that compiles PIR into byte code.
+ +
PBC
+
Parrot Byte Code, the binary form to which all parrot programs are compiled in the end.
+ +
Core setting
+
The core setting is the Perl 6 standard library.
+ +
QAST
+
+ +
JIT
+
+ +
PIRT
+
+
diff --git a/docs/architecture.svg b/docs/architecture.svg index 7e6581e12dd..0d0ab0840a5 100644 --- a/docs/architecture.svg +++ b/docs/architecture.svg @@ -11,13 +11,15 @@ xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" version="1.0" - width="351.00473" - height="429.95145" + width="361.44284" + height="419.12201" id="svg2" sodipodi:version="0.32" inkscape:version="0.48.3.1 r9886" sodipodi:docname="architecture.svg" inkscape:output_extension="org.inkscape.output.svg.inkscape"> + Rakudo Perl 6 Architecture @@ -26,12 +28,34 @@ image/svg+xml + Rakudo Perl 6 Architecture + + + + rakudo architecture + + + + + + Mortiz Lentz + + + + + The Perl Foundation + + + https://raw.github.com/rakudo/rakudo/nom/docs/architecture.svg + https://raw.github.com/rakudo/rakudo/nom/docs/architecture.html + en-US + inkscape:window-maximized="1" + fit-margin-top="5" + fit-margin-left="5" + fit-margin-right="5" + fit-margin-bottom="5" + showguides="true" + inkscape:guide-bbox="true" /> - + + + + + + + id="a66" + transform="matrix(0.5,0,0,0.5,17.715316,-37.789196)"> + Core setting(Perl 6) + + + + The World (NQP) + + + + The Perl 6 source code is transformed in various stages. The first one is the parser stage, which creates a parse tree out of the Perl 6 source code. + +The parser stage is implemented in "Not Quite Perl 6" (NQP) and is part of Rakudo and hosted in the Rakudo repository. + + + Parser (NQP) + + inkscape:connector-curvature="0" /> Parser (P6 Regexes, NQP) + style="font-size:24px">Parser (NQP) - - - - - - - - - - - - - - - - - - - - + + + + The action methods are applied to the parse tree at the same time as the parser builds it. The result of this process is the Abstract Syntax Tree that is sent to the QAST compiler. + Action methods (NQP) - - Perl 6source - syntaxtree - QAST - PIRT - PIR - - - - - + style="fill:#72abff;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 221.18207,386.29776 333.35034,0 0,61.6193 -333.35034,0 z" + id="rect2387" + inkscape:connector-curvature="0" /> PBC - + style="font-size:26.11718369px;font-style:normal;font-weight:normal;fill:#000000;fill-opacity:1;stroke:none;font-family:Bitstream Vera Sans">Action methods (NQP) + + + transform="matrix(0.5,0,0,0.5,12.26668,-98.103967)"> + style="fill:#72abff;fill-opacity:1;fill-rule:evenodd;stroke:none" /> PIRT-compiler (NQP) + x="254.03218" + y="714.18896" + id="tspan2549">PIRT serializer (NQP) + transform="matrix(0.5,0,0,0.5,12.26668,-98.091211)"> + style="fill:#72abff;fill-opacity:1;fill-rule:evenodd;stroke:none" /> QAST-compiler (NQP) + id="tspan2553">QAST compiler (NQP) + transform="matrix(0.5,0,0,0.5,12.266678,-98.116724)"> + style="fill:#00ffff;fill-opacity:1;fill-rule:evenodd;stroke:none" /> IMCC (C) + + Perl 6source + + QAST + + PIRT + + PIR - - Action methods (NQP) - - - - Setting(Perl 6) - - - - - - - - - - - - - - - - - - - - - - - + transform="matrix(0.5,0,0,0.5,12.266687,-98.428182)"> + The POST compiler emits PIR, which IMCC transforms into byte code. IMCC is parrot's PIR compiler, written in C and statically linked into parrot. The byte code (PBC) can then be stored to disk, or executed in memory by a so-called run core or run loop, which is in some sense the heart of parrot - or one of the hearts, because there are several different ones available (one for just-in-time compilation (JIT), one for debugging etc.). + +There are also some supporting custom types and operations in Rakudo called dynamic PMCs and dynamic ops which are written in C, and helper functions written in other languages (namely NQP and PIR). Those do not show up in the flow chart. + Parrot runtime (C) + style="fill:#00ffff;fill-opacity:1;fill-rule:evenodd;stroke:none" /> Parrot runcores (C) + id="tspan2711">Parrot runtime (C) + + PBC - - World, tracks declarations (NQP) + id="g3077-7" + transform="translate(53.63025,218.88296)"> + + + PMC &dynops (C) + + + + + - + id="g6110" + transform="translate(-2.4649472,0)"> + + + + + + + + From 19052593e58e03839d64fdc5c6369e342b8fdd42 Mon Sep 17 00:00:00 2001 From: "Salve J. Nilsen" Date: Wed, 29 Aug 2012 17:35:29 +0200 Subject: [PATCH 2/2] Updated description of the Core settings --- docs/architecture.html | 51 ++++++-------- docs/architecture.svg | 155 ++++++++++++++++++----------------------- 2 files changed, 91 insertions(+), 115 deletions(-) diff --git a/docs/architecture.html b/docs/architecture.html index a6f0f2f2a99..9973fbeea83 100644 --- a/docs/architecture.html +++ b/docs/architecture.html @@ -1,10 +1,10 @@ - Rakudo architectural overview + How Rakudo compiles a Perl 6 program -

Rakudo architectural overview

+

How Rakudo compiles a Perl 6 program

Rakudo architectural overview -

Parser and Action methods

+

Parser and Action Methods

-

The Perl 6 source code is transformed in various stages. The first are the Parser and Action Method stage. This stage creates a parse tree out of the Perl 6 source code and the fires off Action methods that annotate the parse tree, incrementally turning it into an Abstract Syntax Tree (QAST). When an Action Method is done annotating, the control is handed back to the Parser, which then continues parsing the Perl 6 code.

+

The Perl 6 source code is transformed in various stages, of which the first two are the Parser and Action Method stages. The Parser creates a parse tree out of the Perl 6 source code and then gives control to appropriate action methods that annotate the parse tree, incrementally turning it into an Abstract Syntax Tree (AST). When an action method is done annotating, control is handed back to the parser, which then continues parsing the Perl 6 code and "fire off" new action methods as it goes.

-

The result of these two stages is an "improved PAST" (Perl 6 Abstract Syntax Tree) named QAST. This tree is then passed on to the QAST compiler.

+

The result of these two stages interacting is an "improved PAST" (Perl 6 Abstract Syntax Tree) called QAST. This tree is then passed on to the QAST compiler.

-

The Parser and Action methods are implemented in "Not Quite Perl 6" (NQP) and are part of Rakudo and hosted in the Rakudo repository at src/Perl6/Grammar.pm and src/Perl6/Actions.pm.

+

The parser and action methods are implemented in "Not Quite Perl 6" (NQP) and are part of Rakudo and hosted in the Rakudo repository at src/Perl6/Grammar.pm and src/Perl6/Actions.pm.

-

QAST compiler

+

The World

-

The QAST compiler transforms the abstract syntax tree into a PIRT (Parrot Intermediate Representation Tree). In this phase we have the program represented as an AST, and need to translate this into the PIR that the Parrot VM understands. To do this, the QAST compiler does a series of translations on the AST, creating PIRT nodes that implement the operations specified by the QAST nodes.

+

The World is where the parser and the action methods store any declarations they encouter during their runs, including Classes, Types, Signatures, Constants, Subs and Methods.

-

In addition, the QAST compiler is responsible for serializing The World in such a way that the later stages can get access to the declarations stored there during the Parser and Action methods stages.

-

There's also opportunity to apply some VM-specific optimizations at this point. When this is done, the resulting PIRT is passed to the PIRT serializer.

+

QAST compiler

-

This stage is described in nqp/src/QAST/.

+

The QAST compiler transforms the abstract syntax tree into a PIRT (Parrot Intermediate Representation Tree). To do this, the QAST compiler does a series of translations on the AST, creating PIRT nodes that implement the operations specified by the QAST nodes.

-

The World

+

In addition, the QAST compiler is responsible for serializing The World in such a way that later stages can get access to the declarations stored there during the parser and action methods stages.

-

The World is where the parser and the action methods store any declarations they encouter during their runs.

+

There's also opportunity to apply some VM-specific optimizations at this point. When this is done, the resulting PIRT is passed to the PIRT serializer.

+

This stage is described in the different files in the nqp/src/QAST/ directory.

-

PIRT serializer

+

PIRT serializer

-

The PIRT serializer "squashes" the PIR tree into a format that can be passed to Parrot itself and it's IMCC (InterMediate Code Compiler) stage.

+

The PIRT serializer "squashes" the PIR Tree into a format that can be passed to Parrot itself and it's IMCC (InterMediate Code Compiler) stage.

-

You can read more about this at nqp/src/QAST/PIRT.nqp

+

You can read more about this at nqp/src/QAST/PIRT.nqp.

IMCC and Parrot runtime

-

The IMCC (InterMediate Code Compiler) receives the PIR code from the PIRT serializer and then transforms it into Parrot Byte Code (PBC). IMCC is parrot's PIR compiler, written in C and statically linked into parrot. The byte code can then be stored to disk or executed in memory by one of the run cores availabe as part of the Parrot runtime. This is in some sense the heart of Parrot - or one of the hearts; There are several different cores available, includin one for just-in-time compilation (JIT), one for debugging and more.

+

The IMCC (InterMediate Code Compiler) receives the PIR code from the PIRT serializer and then transforms it into Parrot Byte Code (PBC). IMCC is parrot's PIR compiler, written in C and statically linked into parrot. The byte code can then be stored to disk or executed in memory by one of the run cores availabe as part of the Parrot runtime. This is in some sense the heart of Parrot - or one of the hearts; There are several different cores available, including one for just-in-time compilation (JIT), one for debugging and others.

-

You can find out more about this on parrot/docs/imcc/

+

You can find out more about the IMCC in the parrot/docs/imcc/ directory, and about the different run cores in the parrot/docs/running.pod

PMCs and dynops

There are also some supporting custom types and operations in Rakudo called dynamic PMCs and dynamic ops (dynops) which are written in C, and helper functions written in NQP and PIR. These supporting libraries exist for adding features to Parrot that are needed to handle special features in Perl 6.

-

Core setting library

+

Core setting library

-

The part of Rakudo described so far is the stage one compiler. In the build process it is compiled first, and then it compiles the Core setting library down to PBC. "Core setting library" is a fancy term describing the built-in functions which are written in Perl 6, whose purpose is to be the Perl 6 standard library. The result of this compilation is linked together with the stage one compiler and Parrot, the result is the perl6 executable.

- -

The core setting library informs the Parser and Action methods when compiling a Perl 6 program.

+

The core settings library is the library containing the methods, classes and almost all other features that make up the Rakudo Perl 6 implementation. This library is tightly coupled with the perl6 binary, and loaded by default every time perl6 is run.

Glossary

@@ -78,16 +76,13 @@

Glossary

Parrot Byte Code, the binary form to which all parrot programs are compiled in the end.
Core setting
-
The core setting is the Perl 6 standard library.
+
The core setting is the Perl 6 standard library. It is part of the perl6 executable, and contains all the standard features available in Perl 6.
QAST
-
- -
JIT
-
+
The "improved" Abstract Syntax Tree used in Rakudo Perl 6. It contains information about how the program is structured, and what it is supposed to do.
PIRT
-
+
Parrot Intermediate Representation Tree.
diff --git a/docs/architecture.svg b/docs/architecture.svg index 0d0ab0840a5..9844d50688f 100644 --- a/docs/architecture.svg +++ b/docs/architecture.svg @@ -67,8 +67,8 @@ id="base" showgrid="false" inkscape:zoom="1.4142136" - inkscape:cx="237.31829" - inkscape:cy="206.75371" + inkscape:cx="238.0254" + inkscape:cy="213.17704" inkscape:window-x="0" inkscape:window-y="24" inkscape:current-layer="svg2" @@ -80,41 +80,12 @@ showguides="true" inkscape:guide-bbox="true" /> - - - - - - - + id="defs4" /> + target="_top"> (Perl 6) - - - The World (NQP) - + + + + + + The World (NQP) + + + + + id="a61"> The Perl 6 source code is transformed in various stages. The first one is the parser stage, which creates a parse tree out of the Perl 6 source code. @@ -206,9 +187,9 @@ The parser stage is implemented in "Not Quite Perl 6" (NQP) and is par inkscape:transform-center-y="-0.02913637" /> + id="action-methods"> The action methods are applied to the parse tree at the same time as the parser builds it. The result of this process is the Abstract Syntax Tree that is sent to the QAST compiler. <a - xlink:href="architecture.html#post-compiler" - id="a41" + xlink:href="architecture.html#pirt-serializer" + transform="matrix(0.5,0,0,0.5,12.26668,-98.103967)" target="_top" - transform="matrix(0.5,0,0,0.5,12.26668,-98.103967)"> + id="a41"> <rect width="333.35034" height="61.619305" @@ -258,10 +239,10 @@ The parser stage is implemented in "Not Quite Perl 6" (NQP) and is par id="tspan2549">PIRT serializer (NQP)</tspan></text> </a> <a - xlink:href="architecture.html#past-compiler" - id="a46" + xlink:href="architecture.html#qast-compiler" + transform="matrix(0.5,0,0,0.5,12.26668,-98.091211)" target="_top" - transform="matrix(0.5,0,0,0.5,12.26668,-98.091211)"> + id="a46"> <rect width="333.35034" height="61.619305" @@ -281,9 +262,9 @@ The parser stage is implemented in "Not Quite Perl 6" (NQP) and is par </a> <a xlink:href="architecture.html#imcc" - id="a51" + transform="matrix(0.5,0,0,0.5,12.266678,-98.116724)" target="_top" - transform="matrix(0.5,0,0,0.5,12.266678,-98.116724)"> + id="a51"> <rect width="333.35034" height="61.619305" @@ -364,10 +345,10 @@ The parser stage is implemented in "Not Quite Perl 6" (NQP) and is par y="301.65015" id="tspan2527">PIR</tspan></text> <a - xlink:href="architecture.html#parrot-runloops" - target="_top" + xlink:href="architecture.html#parrot-runtime" + transform="matrix(0.5,0,0,0.5,12.266687,-98.428182)" id="a101" - transform="matrix(0.5,0,0,0.5,12.266687,-98.428182)"> + target="_top"> <desc id="desc3042">The POST compiler emits PIR, which IMCC transforms into byte code. IMCC is parrot's PIR compiler, written in C and statically linked into parrot. The byte code (PBC) can then be stored to disk, or executed in memory by a so-called run core or run loop, which is in some sense the heart of parrot - or one of the hearts, because there are several different ones available (one for just-in-time compilation (JIT), one for debugging etc.). @@ -406,37 +387,37 @@ There are also some supporting custom types and operations in Rakudo called dyna id="tspan2541" y="382.36429" x="178.24348">PBC</tspan></text> - <g - id="g3077-7" - transform="translate(53.63025,218.88296)"> + <a + xlink:href="architecture.html#pmc-dynops" + target="_top" + id="a3110"> <g - id="g3059-4" - transform="translate(-32.73544,-1.5968508)"> + id="g3104"> <rect style="fill:#55ff99;fill-opacity:1;stroke:none" id="rect3083-4" width="47.905518" height="103.10033" - x="287.64252" - y="93.73558" /> + x="308.53732" + y="311.0217" /> <text xml:space="preserve" style="font-size:13.05000019px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Bitstream Vera Sans" - x="-145.08183" - y="306.92456" + x="-362.36795" + y="327.81937" id="text3853-9" sodipodi:linespacing="125%" transform="matrix(0,-1,1,0,0,0)"><tspan sodipodi:role="line" id="tspan3855-2" - x="-145.08183" - y="306.92456">PMC &</tspan><tspan + x="-362.36795" + y="327.81937">PMC &</tspan><tspan sodipodi:role="line" - x="-145.08183" - y="323.23706" + x="-362.36795" + y="344.13187" id="tspan3229">dynops (C)</tspan></text> </g> - </g> + </a> <path style="fill:#ffbb55" d="m 304.45517,318.86055 -4.34646,0 -0.008,-4.48275 -7.83783,12.52446 7.89841,12.0777 -0.008,-4.49033 6.86003,0 0,-15.62908 z"