Update Docs

documentation/for-contributors/generators/README.md

@@ -2,12 +2,12 @@
 
 SilkTouch is the Silk.NET project's C# native interoperability code generation stack. It contains useful generators which together generate the majority of the library's codebase.
 
-It does this by splitting the generation process into multiple phases, all anchored around the concept of a "shared infrastructure" - abstractions for SilkTouch generators which are agnostic of where they're running. Currently the only places SilkTouch can run is in a Roslyn Source Generator and using a `dotnet tool` Command Line Interface, but the idea is so long as no incompatibilities are introduced by each individual phase (such as being locked to .NET 6 instead of .NET Standard 2.0) the generator should be able to run in whatever form factor is desirable - the Shared Infrastructure will provide everything a generator could need for consuming a JSON configuration with C# input code and producing additional generated C# code as a result.
+It does this by centering all parts around a [common symbol layer](./symbol-layer/README.md), which allows other parts to be used in any configuration deemed useful, parts don't have a way other then the symbol layer to exchange data, enforcing interoperability.
 
 Learn more about each individual cornerstone of the SilkTouch Stack:
-- [Shared Infrastructure](shared-infrastructure.md)
-- [Emitter](emitter.md)
-- [Overloader](overloader.md)
-- [Scraper](scraper.md)
+
+-   [Symbol Layer](./symbol-layer/README.md)
+-   [Emitter](emitter.md)
+-   [Scraper](scraper.md)
 
 TODO: Add a link to the original proposal once the proposals folder has been reorganised according to the Software Development Plan.

documentation/for-contributors/generators/emitter/README.md

@@ -0,0 +1,7 @@
+# Emitter
+
+The emitter is responsible for creating Roslyn C# Code Symbols from our symbol layer.
+
+See [here](./about%20formatting.md) for how formatting works, and [here](./visitor.md) for details on the visitor implementation.
+
+In general the emitter tries it's best to be very basic doing only the job of 1:1 translating shared symbol layer into C# roslyn symbols.

documentation/for-contributors/generators/emitter/visitor.md

@@ -1,5 +1,7 @@
 # Visitor
 
+See [here](../symbol-layer/symbol-visitor.md) for details on how the base symbol visitor works.
+
 To Transform our internal symbol layer to C# code a visitor (`CSharpEmitter.Visitor`) is used.
 
 ## State

documentation/for-contributors/generators/scraper.md

@@ -1,39 +1,13 @@
 # Scraper
 
-The SilkTouch Scraper is responsible for walking through the abstract syntax tree (AST) of a C/C++ header and all the referenced headers therein to generate C# bindings to as much of this as possible where the configuration indicates this is desirable. Only certain headers will have bindings generated for them, which can be specified by providing the paths to the headers to "traverse" in the JSON configuration.
+The scraper calls into clang to generate XML and can scrape this XML for Symbols of the [Symbol Layer](./symbol-layer/README.md)
 
-## ClangSharp
+## Scraping XML
 
-To walk through the C/C++ AST, ClangSharp's P/Invoke Generator is used, which is a proven tool for generating C# bindings to C/C++ APIs published by Microsoft. It does this by using the clang compiler to walk through the AST.
+To scrape XML there is a visitor that visits each XML node and outputs several symbols per node.
+This is what happens in `XmlVisitor.cs`
 
-ClangSharp's P/Invoke Generator is capable of producing C# bindings on its own, however these are raw bindings that use DllImport and don't aim to lift the bindings to be more C#-friendly. However, as part of Scraper development work was done to add an "XML output mode" to ClangSharp which instead of producing pure C# code, it outputs XML which roughly represents what the C# code would've looked like.
+## Generating XML using clang
 
-## The Subagent
-
-ClangSharp uses libclang, which has funny rules over multithreading. To maintain a degree of parallelism during generation, the invocation of ClangSharp was moved to be done in a separate process. The original intention was to invoke the `ClangSharpPInvokeGenerator` tool as shipped publicly, but unfortunately `dotnet tool`s cannot be referenced by other libraries or `dotnet tool`s. As such, we created our own shim which mimics the behaviour of and duplicates a lot of code from the `ClangSharpPInvokeGenerator` tool, but within the same CLI executable.
-
-### The `ISubagent` abstraction
-
-`ISubagent` is responsible for launching the ClangSharp tool with certain parameters, inputs, and configuration. This is an abstraction such that the main Scraper assembly remains easily portable to another form factor should we enable the Scraper to run in another form factor in the future, as well as to prevent the library assembly knowing too much about the assembly in which its executing.
-
-The only implementation of this interface is the `ClangSharpSubagent` class in the `SilkTouch` assembly (CLI implementation of the Shared Infrastructure) which gathers information on the currently executing assembly, and relaunches it with `clangsharp` as the first argument and a JSON-encoded record as the second.
-
-### The `silktouch clangsharp` command
-
-This is the command that invokes ClangSharp as described in the previous section. This deserializes the JSON-encoded record and passes the data held therein to the `PInvokeGenerator` class within ClangSharp. This will generate the XML dump and then quit, possibly with some ClangSharp-generated diagnostics and a non-zero error code if those diagnostics are severe. The implementation of this is in the `ClangSharpHandoff` class.
-
-## XML Mods
-
-Once an initial XML dump has been generated by The Subagent, the XML dump will be parsed by the `Silk.NET.SilkTouch.ClangSharp.Xml` library and produce a tree of records. This is done in the `ScraperGenerator` ahead of the XML mods stage.
-
-The purpose of the XML mods stage is to walk this tree of records, and modify it depending on the project configuration (for example changing all names to be PascalCase, or adding custom attributes to cause the Overloader to generate certain overloads). Each mod must be specifically enabled, and has free range over the XML. As such, they execute in the order in which they were specified in the project configuration.
-
-Each XML mod has access to the `ScraperJobConfiguration.ModOptions` and the full XML tree. It does not have any access to resources beyond that, such as the SilkTouch Context.
-
-TODO: Write words here about each XML mod once XML mods have been implemented
-
-## Transformation
-
-After the XML mods have been applied, the transformation stage is responsible for converting the bindings represented by the XML tree back into C# code that can be output via the SilkTouch Context (and added to the project compilation). Unlike ClangSharp, this stage will output bindings which are compatible with/invoke the Emitter and Overloader instead of using DllImport directly.
-
-TODO: Write more words here once this stage is implemented.
+This is what happens in `ClangScraper.cs`, which is configured using `ClangScraperConfiguration.cs`
+Check XML docs for details on what methods do.

documentation/for-contributors/generators/symbol-layer/README.md

@@ -0,0 +1,11 @@
+# Symbol Layer
+
+The symbol layer is a set of types in the Silk.NET.SilkTouch.Symbols, all inheriting from the central Silk.NET.SilkTouch.Symbols.Symbol type.
+
+The symbol layer is intended as a way to represent data passed between components of SilkTouch and is entirely immutable, with the only exception being the [Type Store](./type-store.md).
+
+The way to update & iterate symbols is using a [Symbol Visitor](./symbol-visitor.md).
+
+See [symbols](./symbols/README.md) for a list of all symbols, their visitor method, test details, etc.
+
+See [type references](./type-references.md) for an explanation of type references.

documentation/for-contributors/generators/symbol-layer/symbol-visitor.md

@@ -0,0 +1,19 @@
+# Symbol Visitor
+
+A symbol visitor visits each symbol and then calls the appropriate methods to recursively visit all parts.
+Each symbol visitor has access to a [type store](./type-store.md). It has to be provided in the constructor of every symbol visitor.
+It's generally adviced to simply bubble up this constructor parameter to users of your type.
+
+## Updating Symbols
+
+Each method returns the same type it gets as parameter, so prefer to override the most specific method possible.
+For example, if you wish to rename all `TypeSymbol`s, override `VisitTypeSymbol`. But if you want to add a field to every struct, override `VisitStructSymbol`. If you want to rewrite the type of a symbol, for example generate a class for every struct, again, override the most specific method that is compatible with both (in this case `VisitTypeSymbol`) and do type checks as necessary.
+
+### Type IDs
+
+See [type store](./type-store.md) for details on what type Ids are.
+**Do not change type IDs when visiting symbols. This breaks all references to that type.**
+
+## Managing State
+
+While the base `SymbolVisitor` is stateless, derived types are free to introduce (mutable) state. This allow propagating state other then the new symbol up the tree.

documentation/for-contributors/generators/symbol-layer/symbols/README.md

@@ -0,0 +1,51 @@
+# Symbols
+
+This document / folder is for tracking all the available symbols, and documenting what creation of one entails.
+
+## Relation to SymbolVisitor
+
+In general each symbol listed below should have a corresponding method `VisitMySymbol` the only exception being `UnresolvedTypeReference` as only few visitors should ever interact with it.
+
+## List
+
+(Order alphabetically please!)
+
+Parent Symbols (Unlisted, abstract):
+| Name | See Also |
+| ---- | -------- |
+| MemberSymbol | |
+| Symbol | |
+| TypeReference | [here](../type-references.md) |
+| TypeSymbol | |
+
+| Name                    | Symbol Layer File                                                                         | Symbol Layer Tests                                                                                             | Emitter Tests                                                                          |
+| ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
+| ExternalTypeReference   | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/ExternalTypeReference.cs) | [here](../../../../../tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/ExternalTypeReferenceTests.cs) | TODO!!!                                                                                |
+| FieldSymbol             | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/FieldSymbol.cs)           | [here](../../../../../tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/FieldTests.cs)                 | [here](../../../../../tests/Silk.NET.SilkTouch.Emitter.Tests/EmitterFieldTests.cs)     |
+| IdentifierSymbol        | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/IdentifierSymbol.cs)      | [here](../../../../../tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/IdentifierTests.cs)            | TODO!!!                                                                                |
+| InternalTypeReference   | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/InternalTypeReference.cs) | TODO!!!                                                                                                        | TODO!!!                                                                                |
+| NamespaceSymbol         | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/NamespaceSymbol.cs)       | [here](../../../../../tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/NamespaceTests.cs)             | [here](../../../../../tests/Silk.NET.SilkTouch.Emitter.Tests/EmitterNamespaceTests.cs) |
+| PointerTypeReference    | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/PointerTypeReference.cs)  | [here](../../../../../tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/PointerTypeReferenceTests.cs)  | TODO!!!                                                                                |
+| StructSymbol            | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/StructSymbol.cs)          | [here](../../../../../tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/StructTests.cs)                | [here](../../../../../tests/Silk.NET.SilkTouch.Emitter.Tests/EmitterStructTests.cs)    |
+| UnresolvedTypeReference | [here](src/generators/Silk.NET.SilkTouch.Symbols/UnresolvedTypeReference.cs)              | [here](tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/UnresolvedTypeReferenceTests.cs)              | -                                                                                      |
+
+## How to create a symbol
+
+Checklist:
+
+-   [ ] Write a short description of what this symbol does in the above list.
+-   [ ] Add the symbol to Silk.NET.SilkTouch.Symbols
+-   [ ] Add a partial of SymbolVisitor to below the definition, and add visiting method in there.
+-   [ ] Call the new visiting method either from
+    -   The root Visit(Symbol) method, this should rarely be needed
+    -   The parent's visit method
+-   [ ] Add Tests to Silk.NET.SilkTouch.Symbols.Tests for
+    -   [ ] The type being visited with it's own visiting method
+    -   [ ] The type being visited with it's parent (not needed if this is a new root type, with no parent)
+    -   [ ] Each part being visited correctly. Only shallowly check this, don't check parts of parts.
+-   [ ] Handle symbol in Silk.NET.SilkTouch.Emitter
+-   [ ] Add Tests to Silk.NET.SilkTouch.Emitter.Tests
+    -   [ ] Transforming a sample Symbol matching a specific string
+    -   [ ] Other tests, this is highly dependend on how complex the C# output looks like
+    -   [ ] More is better!
+-   [ ] You're done. Feel free to use the symbol wherever needed. Prefer to add usages in a separate PR.

documentation/for-contributors/generators/symbol-layer/type-references.md

@@ -0,0 +1,14 @@
+# Type References
+
+Type references are a special kind of symbol used to reference types.
+
+There are several kinds of type references.
+During scraping (see [scraper](../scraper.md)) `UnresolvedTypeReference`s are created.
+These should be resolved right after, in a process called [type resolution](../type-resolution.md).
+
+Type resolution leads to a variety of more specialized type references, with the two base ones being
+
+-   `InternalTypeReference`, referencing another newly defined type somewhere in the tree. (Note that it's generally not adviced to define types this way, they should already occur somewhere higher up in the tree)
+-   `ExternalTypeReference`, referencing some external type, usually well known types from the standard library
+
+There are several other type reference types, but they are wrappers around other references, for example `PointerTypeReference`.