Guided Aardvark.Media Walkthrough
- What is aardvark media?
- Aardvark.Media is a functional programming interface to build applications without additional complexity (boilerplate) for handling change, i.e. synchronizing data and it's visual representations, while providing high performance adaptive 3D rendering (via Aardvark.Rendering) and incremental HTML generation (similar to react).
- What does it do?
- It tames complexity by enforcing the unidirectional data flow paradigm on programmers. This architecture known as the ELM architecture (model/update/view) allowing them to build large complex apps from smaller apps through composition, while each app can be tested independently. Several optimization mechanisms in the background are employed to let programmers focus on actual functionality instead of performance details leading to cleaner, conciser, more robust, and scalable code.
- Aardvark.Media offers a seamless combination of HTML GUI elements and 3D real-time rendering nodes which are inherently consistent with the data model. Changes to the data result in incremental changes of the DOM tree and the 3D scene graph, i.e. only relevant parts are updated.
- What it doesn't do?
- It does not redeem programmers from thinking ;) ... but you typically have to reason about a problem once, solve it and it stays that way.
- It is not a rendering engine offering a complete set of built-in features, such as collision detection, tone mapping, or 3D transform controls. However, we provide a programming environment and paradigm which allows you to build such components in a robust and concise way helping you to enrich the Aardvark ecosystem.
All components of the Aardvark ecosystem are mainly programmed in F#, a functional programming language. Hence, it is often said one has to 'unlearn' imperative programming styles and object-oriented thinking. F# for Fun and Profit gives a great and complete overview of F# and also elaborates on why to use it. This is not about learning a new language, this is about evolving a new way to think, a new way to approach problems, and only ultimately expressing this in a new programming language.
Good starting points are the following two talks functional design principles and a functional approach to domain driven design. After getting an idea about the functional paradigms you can dive more into the syntax and language functions covered in the written series.
For programming web-based GUI elements within the Aardvark platform, it is important to know your way around web technologies, such as, HTML, CSS, SVG, and JavaScript. There is no really useful complete tutorial resource for our needs, so we will explain these parts by example in later sections. For most of our visible UI parts we found Semantic UI very useful, but you could use any other UI library together with Aardvark.Media.
The execution model of every App in Aardvark.Media follows the same pattern of unidirectional data flow, known as the ELM architecture (see ELM). It is most natural to start with the Model, which shall hold our data. The View describes how this data is visualized. In Aardvark.Media this is either a DomNode (part of an HTML page) or a SceneGraph node for 3D real-time rendering. The user can interact with what the view produces, e.g. clicking on a button or picking a 3D object, which in turn can generate Actions. These actions strive to change our data, e.g. the color of an object the user selects in a colorpicker. The Update logic specifies how these actions change our data and generate a new model (immutable data). What we call an App is simply a combination of Model-Update-View and its Actions.
Apps exist at different granularities. There is an app for a numeric data field while a whole project is also just and app. Larger, more complex apps are simply composed together from smaller apps. It is quite natural that complex model types consist of more primitive model types and ultimately primitive datatypes on the lowest level. In later sections we will see how views, actions and update functions can easily be composed together to build powerful and maintainable apps.
If you have not done it already for some other Aardvark tutorial, just perform the following steps.
> git clone https://github.com/aardvarkplatform/aardvark.docs.git
> cd aardvark.docs
> build
Then just open src\aardvark.docs.sln with visual studio. The demos discussed in the following Chapters can be found in the ModelUpdateView project. If you want to start your own Aardvark project or Aardvark.Media project just clone our template...
> git clone https://github.com/aardvark-platform/template.git
> cd template
> build
... and follow the instructions in the command window. Please be sure to push modifications into your own repo ;)
The following example shall lead us through all topics relevant to building powerful apps in Aardvark.Media combining 3D graphics and a web-based GUI in a declarative and functional way. We will render a couple of 3D objects and transform them with controls we write from scratch. We demonstrate how the Model-Update-View paradigma works for generating GUI and 3D views and how to build powerful apps by composition. In the later chapters we we will demonstrate how our incremental evaluation system and its adaptive data structures handle change and are essential to tame complexity in larger apps.
First, we will build a numeric control, which lets users increment and decrement a float value. We will approach this problem in the sequence of Model-Update-View, starting with the model.
We simply build a record type holding our float value. For future use, we also attribute it with ModelType.
[<ModelType>]
type Model = { value : float }Specifying update is a two-fold effort of first defining actions of what does our app do to the model ...
type Action =
| Increment
| Decrement... and second of defining how do these actions affect our model.
let update (m : Model) (a : Action) =
match a with
| Increment -> { m with value = m.value + 0.1 }
| Decrement -> { m with value = m.value - 0.1 }The typical signature of update is Model -> Action -> Model, taking the current model and an action and returning an updated version of our model. Although, actions can in principle come from anywhere our actions for now will originate from our app's UI, which we specify in the view function. Consequently, the view function defines how our model is visualized and how the users can manipulate the model via UI elements.
First, we create two buttons , one for incrementing and one for decrementing our model's value.
button [clazz "ui button"; onClick (fun _ -> Increment)] [text "+"]
button [clazz "ui button"; onClick (fun _ -> Decrement)] [text "-"]Besides the architecture of ELM we were also inspired by ELM's way of constructing DOM (Document Object Model) elements, such as divs, buttons, and unordered lists. This allows us to emit HTML code for our views directly from F# code. The code above roughly equivalents to:
<button class="ui button" onClick=Increment()>+</button>
<button class="ui button" onClick=Decrement()>-</button>You may notice that we write the class attribute as clazz. This is not because we are the cool kids, but naturally class is a reserved keyword in F#. Most of the HTML and Aardvark.UI tags are elems consisting of an attribute list and a child list, while voidelems such as br only have attributes. A list of all available tags can be found here. The finished view function, which also contains a visualization for our model, may look like the following:
let view (m : AdaptiveModel) =
require Html.semui (
body [] (
[
div [] [
button [clazz "ui button"; onClick (fun _ -> Increment)] [text "+"]
button [clazz "ui button"; onClick (fun _ -> Decrement)] [text "-"]
br []
text "my value:"
br []
Incremental.text (m.value |> AVal.map(fun x -> sprintf "%.1f" x))
]
]
)
)require Html.semui enables the emitted HTML code to refer to web resources such as CSS style sheets or JavaScript libraries. In this case we emit a script reference for the UI styling library Semantic UI. For now we ignore AdaptiveModel, Incremental.text, and AVal.map which will be discussed in the next section.
The final lines of every app are connecting the individual parts:
let app =
{
unpersist = Unpersist.instance //ignore for now
threads = fun _ -> ThreadPool.empty //ignore for now
initial = { value = 0.0 }
update = update
view = view
}
let start() = App.start app...resulting in the following output:
Our vector control shall allow us to manipulate XYZ of a 3D vector. Naturally, we want to reuse our previous example and just combine three numeric controls via composition. First, we create a composed VectorModel by using a record type consisting of individual NumericModels for x,y and z.
[<ModelType>]
type VectorModel = {
x : NumericModel
y : NumericModel
z : NumericModel
}Our composed app now has a composed model and of course needs a composed action describing what our VectorControl can do. If we come across an action in this control either x,y or z is incremented or decremented. However, increment and decrement are already captured in the NumericControl.Action, so we don't want to specify that again. Since an action is either an update on x or on y or on z our composed action is the following union type.
type Action =
| UpdateX of NumericControl.Action
| UpdateY of NumericControl.Action
| UpdateZ of NumericControl.ActionLooking at Model and Action, our VectorControl is quite limited, it can't do anything by itself really. Therefore, our update function just simply calls update functions from the NumericControl.
let update (m : VectorModel) (a : Action) =
match a with
| UpdateX a -> { m with x = NumericControl.update m.x a }
| UpdateY a -> { m with y = NumericControl.update m.y a }
| UpdateZ a -> { m with z = NumericControl.update m.z a }The results are handed to the relevant part of our composed model, e.g. the x coordinate. The match on UpdateX unpacks the composed action a and we can directly pass a NumericControl.Action to update. Update returns a new NumericModel which we then use to update the VectorModel.
Such pure composition apps are quite frequent, for instance when making a property control consisting of numeric, bool, and string values, or when making a sidebar app consisting of multiple property apps. But more often composing apps add their own actions. In this case we could add Normalize and Reset to our VectorControl.
type Action =
| UpdateX of NumericControl.Action
| UpdateY of NumericControl.Action
| UpdateZ of NumericControl.Action
| Normalize
| Reset
let update (m : VectorModel) (a : Action) =
match a with
| UpdateX a -> { m with x = NumericControl.update m.x a }
| UpdateY a -> { m with y = NumericControl.update m.y a }
| UpdateZ a -> { m with z = NumericControl.update m.z a }
| Normalize ->
let v = V3d(m.x.value,m.y.value,m.z.value)
v.Normalized |> toVectorModel
| Reset -> VectorModel.initialBefore starting with the view, we will adapt the viewing code of the NumericControl a bit. In the way we programmed it above it is not very suitable to be stacked together into a VectorControl. That being said, a composing app is responsible for how the composed parts should be viewed together.
let view' (m : AdaptiveNumericModel) =
require Html.semui (
table [][
tr[] [
td[] [a [clazz "ui label circular Big"] [Incremental.text (m.value |> Mod.map(fun x -> " " + x.ToString()))]]
td[] [
button [clazz "ui icon button"; onClick (fun _ -> Increment)] [text "+"]
button [clazz "ui icon button"; onClick (fun _ -> Decrement)] [text "-"]
]
]
]
)This is our updated view function view', which uses an HTML table to align things more neatly. The result looks like this:
Our VectorControl's view function should now call the individual NumericControl view functions and put them in a nice table layout.
let view (m : AdaptiveVectorModel) =
require Html.semui (
div[][
table [] [
tr[][
td[][a [clazz "ui label circular Big"][text "X:"]]
td[][NumericControl.view' m.x |> UI.map UpdateX]
]
tr[][
td[][a [clazz "ui label circular Big"][text "Y:"]]
td[][NumericControl.view' m.y |> UI.map UpdateY]
]
tr[][
td[][a [clazz "ui label circular Big"][text "Z:"]]
td[][NumericControl.view' m.z |> UI.map UpdateZ]
]
tr[][
td[attribute "colspan" "2"][
div[clazz "ui buttons small"][
button [clazz "ui button"; onClick (fun _ -> Normalize)] [text "Norm"]
button [clazz "ui button"; onClick (fun _ -> Reset)] [text "Reset"]
]
]
]
]
]
)Everything looks quite familiar, except for UI.map. The view function of the VectorControl is supposed to return DomNode<Action>, while NumericControl.view returns a DomNode<NumericControl.Action> resulting in a type mismatch. UI.map resolves this problem by lifting the action type of the sub app to the action type of the composing app by simply mapping it to the respective composing action.
The deep nesting of DOM nodes results from the quite cumbersome way to specify tables in HTML. Further, the vector control is probably not the most elegant control you have ever seen. However, we will stick to the this level of verbosity for explanatory reasons. Using the UI primitives of Aardvark.Media results in something more like this:
Html.SemUi.accordion "Scale" "Expand" true [
div [] [Vector3d.view model.scale |> UI.map V3dMessage]
]
In the described implementation we used composition to build a vector control by just reusing our numeric control code. In the same way we can compose a transformation app for manipulating 3 vectors for translation, rotation, and scale. Instead of implementing this step by step we use it to revisit the composition illustration from earlier filling it with actual code.
In this chapter we covered how the meaning of model-update-view and what apps look like in Aardvark.Media. We built a small app from scratch and composed a more powerful app from that. We further illustrated how to use domnodes which ultimately emit html and javascript code for our GUI.
Before expanding our example with a 3D view we will revisit the ELM architecture and cover incremental evaluation.
When naively implementing the ELM architecture illustrated above we execute the whole view function on every update of the model. This does not scale well with larger DOM trees, larger SceneGraphs, nor high frequency updates, such as camera movement. We actually want to only update those parts of the scene which have actually changed - so we want incremental updates.
Aardvark.Rendering is a powerful rendering engine which supports incremental updates of 3D scenes, while Aardvark.Media does the same thing for DomNodes. We will not go into details here (refer to publications) but at the heart of all this is the Adaptive System. Instead of specifying on how to react to change explicitly a dependency is created between a data field in the model and a visual representation, e.g. a color : C4b and the rendering color of an Sg.Cube.
Within our nice unidirectional dataflow we don't want to use the mod system directly. Therefore, a compile step generates an AdaptiveModel from our Model, translating every data field to an AVal field which is aware of being changed. Most Sg nodes take AVal versions and for GUI elements there are incremental tags, such as Incremental.Text taking an AVal<string>.
Here the incremental loop is illustrated with actual code. Our model contains a 3D vector (V3d) specifying a scale. Our compiler, the diff generator, generates an AdaptiveModel wrapping the V3d into a mod. Our 3D view shows a cube which is transformed and further triggers a reset action on double click. Sg.trafo takes AVal<Trafo3d>. To generate an AVal<Trafo3d> we use Mod.map and specify a mapping AVal<V3d> -> AVal<Trafo3d>. The Sg view and the DomNode view can both trigger a reset action which changes our model. This change is reflected in an update of AdaptiveModel.scale which only triggers a reevaluation of Sg.trafo.
In the previous Section we only dealt with very limited scene modifications, namely the change of a single value or property e.g. the scale of a box. In this section we will focus on how to handle structural changes to our scene. Our leading example will be the BoxSelectionDemo, availabe in Aardvark.Media when loading the Aardvark.Media.Examples.sln` solution. Our application can add and remove boxes and further offers interaction to select and hover boxes in 3D as well as in a GUI list view. Adaptive datastructures are crucial for dealing with structural change, but we will also cover other relevant topics along the way, such as integrating a camera controller.
When recalling the previous chapter, we got to know two types that 'react to change', AVal<'a> for wrapping a value into an AVal and the adaptive type AdaptiveType provided by the diffgenerator through the [ModelType] attribute. Since functional programming is all about data structures we also need adaptive versions of collections such as lists, sequences, or maps.
| Immutable | Adaptive | Remarks |
|---|---|---|
α(IndexList<'T>) |
alist<α('T)> |
alist<aval<'T>> -> alist<'T>
|
α(HashSet<'T>) |
aset<α('T)> |
aset<aval<'T>> -> aset<'T>
|
α(HashMap<'K, 'V>) |
amap<'K, α('V)> |
amap<'K, aval<'V>> -> amap<'K, 'V>
|
α({ a:'T;.. }) |
{ a:α('T);..} |
product types |
α('T * ..) |
(α('T) * ..) |
tuples |
α(struct('T * ..)) |
(α('T) * ..) |
struct tuples |
α(|A of 'T..) |
aval<|AdaptiveA of α('T)..> |
sum types |
α('T) |
aval<'T> |
opaque types |
This table shows the compilation scheme of the adaptify service, telling us which type is mapped to which adaptive type, for instance, an IndexList in our domain type Abc will be an alist in our adaptive type AdaptiveAbc. Adaptive datastructures are part of FSharp.Data.Adaptive and the adaptify service is creating them from immutable types. Both links contain extensive example and tutorial ressources supporting a deeper understanding of these mechanics.
Back to our BoxSelectionApp, we start again with the model
[<ModelType>]
type BoxSelectionDemoModel = {
boxes : IndexList<Box3d>
}containing an alist of Box3d. In AdaptiveBoxSelectionDemoModel this will be accessible for our view function as alist<Box3d>. We want our 3D view and our GUI, a list of boxes, to react to changes of this list, e.g. add a new 3D cube if a box is added. alist not only reacts to structural changes, but also to changes to the properties of each element efficiently propagated via the dependency graph.
[<ModelType>]
type VisibleBox = {
geometry : Box3d
color : C4b
[<NonAdaptive>]
id : string
}
[<ModelType>]
type BoxSelectionDemoModel = {
camera : CameraControllerState
boxes : IndexList<VisibleBox>
boxHovered : option<string>
}To make our naked box geometries a bit more interesting we wrap them into VisibleBox which also holds a color and an id. The attribute [<NonAdaptive>] tells adaptify to not compile this field to an AVal. Of course, we want our ids constant, otherwise it will be difficult to identify our boxes.
As you can see, we also included a CameraControllerState into our model. To have an interactive 3D view, we simply compose a CameraControllerApp into our app starting with just adding its model, which should rather be called CameraControllerModel if we were a 100% strict with the naming. Next, we wrap the camera controller actions into our BoxSelection actions and handle the action in the update function, as with any other composition.
type Action =
| CameraMessage of CameraControllerMessage
...
let update (model : BoxSelectionDemoModel) (act : Action) =
match act with
| CameraMessage m ->
{ model with camera = CameraController.update model.camera m }Finally, we embed the CameraController into our view function. The CameraController's view is exposed as an incremental control. On second thought, this is just another view function taking a model, an action, and some additonal attributes. Most notable here are the AttributeMap and the child nodes as last argument.
let view (model : AdaptiveBoxSelectionDemoModel) =
let frustum = Mod.constant (Frustum.perspective 60.0 0.1 100.0 1.0)
div [clazz "ui"; style "background: #1B1C1E"] [
CameraController.controlledControl model.camera CameraMessage frustum
(AttributeMap.ofList [
attribute "style" "width:65%; height: 100%; float: left;"
])
(
Sg.box (AVal.constant C4b.Gray) (AVal.constant Box3d.Unit)
|> Sg.shader {
do! DefaultSurfaces.trafo
do! DefaultSurfaces.vertexColor
do! DefaultSurfaces.simpleLighting
}
|> Sg.requirePicking
|> Sg.noEvents
)
]AttributeMaps are a special way of defining changeable attributes for a DomNode. In the above case we us static attributes for an incremental DomNode, so we can just use static syntax and convert it to an AttributeMap via AttributeMap.ofList. The child node parameter of the controlledControl function is of the type ISg<'msg>, in our case a static grey box not capable of triggering any actions. Most interestingly the output of controlledControl is a DomNode of Action, since the 3D rendering frames are written into an image. Therefore, the result of our controlledControl function composes with any other DomNodes, just as a div or a text would. Our interactive intermediate result looks like this:
Before dealing with multiple boxes we think of the actions we want to support and implement them for a single box.
type Action =
| CameraMessage of CameraControllerMessage
| Select of string
| ClearSelection
| HoverIn of string
| HoverOut
| AddBox
| RemoveBox We want to click on a box and Select it resulting in a persistent highlighting (color red) or a peek highlighting on HoverIn (color blue) which stops on HoverOut. Similar to attaching actions to DomNodes (onclick button) we can specify which actions are triggered when clicking on a Sg.Box.
Sg.box color box.geometry
|> Sg.shader {
do! DefaultSurfaces.trafo
do! DefaultSurfaces.vertexColor
do! DefaultSurfaces.simpleLighting
}
|> Sg.requirePicking
|> Sg.noEvents
|> Sg.withEvents [
Sg.onClick (fun _ -> Select box.id)
Sg.onEnter (fun _ -> Enter box.id)
Sg.onLeave (fun () -> Exit)
]Since we need to know which box has been selected/hovered later on we send the box.id with the Action into the update function. For simplicity we will we will only go through the hovering interaction here.
let update (model : BoxSelectionDemoModel) (act : Action) =
match act with
| CameraMessage m ->
{ model with camera = CameraController.update model.camera m }
| HoverIn id -> { model with boxHovered = Some id }
| HoverOut -> { model with boxHovered = None }
| _ -> model
We map our hovering interaction to the option of boxid. Either, there is an id or no hovered has taken place. Now we want to react on this changing hovering state by changing the cubes color. adaptify gives us an AVAl<Option<string>> in the view function, while we need an AVal<C4b>. Once again, we want to just map things from AVal<'a> to AVal<'b> using AVal.map.
let color =
model.boxHovered
|> Mod.map (fun x ->
match x with
| Some k -> if k = "box" then C4b.Blue else C4b.Gray
| None -> C4b.Gray)This last Chapter is a quick reference for the reactive cells "AVals". Please also read the full documentation and tutorials here.
AVal versions of our values are part of the dependency graph constructed and maintained in the background. Each AVal is aware of its own change and the change of its dependencies. Without going into much detail regarding the theory behind this, the AVal system is the very backbone of Aardvark.Media ensuring that when we change our model only the relevant parts of our views are updated... automatically, always!
The data in our model and the data needed for a view are rarely directly compatible. For intance, we want to highlight selected boxes in a certain color, our data model has an isSelected state of bool, or AVal as part of the MModel, but our Sg.box needs an aval so we can actually influence it's color. To achieve this we simply map one AVal to the other.
AVal.map f x
f('a -> 'b) -> aval<'a> -> aval<'b>
let color =
m.isSelected
|> AVal.map (fun x ->
if x then C4b.red else C4b.white
)We can think of AVal as boxes. We take type 'a out of the box, modify it with f to be of type 'b and then put it back into the box.
Often we want to use a value in our mappings that is already a AVal. For example, the selection color is not just fixed at C4b.red, but also depends on the state of our domain model. These are called "structural dependencies".
let selectionColor =
AVal.constant C4b.Red
let color =
m.isSelected
|> AVal.map (fun x ->
if x
then selectionColor
else Mod.Constant C4b.white
)
//returns AVal<AVal<C4b>>Color now is of type aval<aval>. We can't really work with that. To solve this elegantly, we have AVal.bind.
AVal.bind f x
f('a -> aval<'b>) -> aval<'a> -> aval<'b>
let color =
m.isSelected
|> AVal.bind (fun x ->
if x
then selectionColor
else AVal.Constant C4b.white
)
//returns AVal<C4b>Recalling the box analogy, you can simply think of having a box inside a box. AVal.bind just unpacks the inner box and 'a lands in the outer box before it is transformed by f like in AVal.map.
AVal.map2 f x y
f('a -> 'b -> 'c) -> aval<'a> -> aval<'b> -> aval<'c>The adaptive computation expression builder exposes AVal.bind as let!
adaptive {
let! a = a
let! b = b
return f a b
}These builders expose adaptive collection types. AVal can be unpacked with let!, while values are enumerated with yield. Please also read the full documentation for more infos on the collection types.
alist,amap,aset {
yield ...
}As you might have already guessed, all these operations or transformations specify edges in our dependency graph and can be chained together naturally. The change of 'a in a AVal.map triggers f to produce a 'b. Consequently, the way you structure your dependency graph through the use of map, bind, adaptive blocks or data structures has an influence on which parts are updated and which fs have to be recomputed.
This can have implications on the performance as we have seen in adaptive blocks.
Some concepts are showcased in the BoxSelectionDemo. Also have a look at the other examples in that repository.
Lenses can be a good fit for Aardvark.Media projects. https://xyncro.tech/aether/guides/lenses.html