Overlay manager discussion thread

[originalfoo]

Dsicussion & brainstorming of overlay manager.

This is what I'm currently chipping away at:

Start

Overlay session starts in one of two ways:

• TurnOn()
• Automatic (infoview, toolcontroller)

If an existing session is running, flag "diffs" between current and new session allow some of the existing caches/overlays to be reused and/or updated, thus minimising workload.

The process creates two structs (which are passed to other stages via ref):

• RenderConfig - defines which overlays to show -> what to map cache, etc.
• RenderState - current state (eg. camera, mouse, keyboard, etc)

Map Caches

Adapted from Krzychu124 code, these use game's render grids to quickly determine what's on-screen and cache ids of validated nodes, etc.

What gets scanned is determined by RenderState which in turn is derived from RenderConfig. The results are cached until told otherwise - this means that if the camera isn't moving, the map cache is idle (we don't need to keep checking what's on-screen).

Because we know which overlays require which caches (target flags per overlay) the generator is able to invoke the required generators.

Additionally, it means we can do targeted micro-generations if new stuff is added to map (eg. Move It pastes intersection) without even needing to refresh the map caches.

Generators

Each overlay has a generator. The generator is responsible for iterating the supplied information to determine what is of interest, and then instantiating wrapper objects of those items, which are then handed to, and cached by, one or more overlay layers.

Code example: Node labels generator

        public override void OnCameraMoved(ref OverlayConfig config, ref OverlayState state) {

            var nodes = ViewportCache.Instance.visibleNodes;
            var numItems = nodes.Count();

            LabelLayer.Instance.MakeRoomFor(numItems);

            for (int i = 0; i < numItems; i++)
                LabelLayer.Instance.Add(new Label(nodes.m_buffer[i]));
        }

        public override void OnModifierChanged(ref OverlayConfig config, ref OverlayState state) {
            if (state.Shift || permaDetail) {
                permaDetail = state.Shift;
                LabelLayer.Instance.Invalidate(this.Overlay);
            }
        }

You can also see in OnModifierChanged() method how the overlay can invalidate all it's labels. In this case holding shift changes what text is returned from the Label.GetText() method.

Layers

Not shown in diagram above.

Each type of overlay (connector, icon, label, etc) has an associated layer. When a generator passes an item to the layer, the layer wraps it in a render cache struct (a "task") and adds that to its 'task list'.

The layer provides tools for updating/invalidating/clearing the contents of the task list.

Code example: Some features of Label layer

        internal void MakeRoomFor(int numItems) =>
            tasks.MakeRoomFor(numItems);

        internal void Add(ILabel label) => tasks.Add(Task.For(label));

        internal void Clear() => tasks.Clear();

        internal void Remove(Overlays overlay) {
            for (int i = tasks.Count(); i-- > 0;)
                if (tasks[i].Label.Overlay == overlay) tasks.RemoveAt(i);
        }

        internal void Invalidate() {
            for (int i = 0; i < tasks.Count(); i++) {
                ref var task = ref tasks.m_buffer[i];
                task.IsDirty = true;
                task.Save(ref tasks.m_buffer, i);
            }
        }

I did add a bunch of extension methods for FastList to make working with them more pleasing. :)

Interact

This is always processing per frame, but is very fast. It iterates the tasks (structs - on heap) and checks task.BoundsBox.Contains(mousePos) - only if that's true does any additional work get done - eg. triggering OnHover() or OnClick() events of the task.Item instance.

Code example: Label interactions

The following are cached values in the task struct (on heap):

• Bounds - bounds box of the label
• IsHovered - current hover state

Task.Label is a class instance (not on heap) so we avoid touching it unless there is an interaction. If interactions occur, the Task.Label methods can return true to invalidate the task render cache (it will be refreshed next Compute step).

If anything in the task struct changes, it will be marked Altered. If Altered, the exissting element in task list is overwritten to store changes.

                bool hovered = task.Bounds.Contains(state.MousePos);

                if (hovered && task.Label.IsInteractive) {
                    task.Interact(hovered, ref state);
                } else if (task.IsHovered) {
                    task.Interact(false, ref state);
                }

                if (task.Altered) task.Save(ref tasks.m_buffer, i);

Compute

This is the most intensive stage. It iterates the task list structs, and if any are dirty (invalidated) it does whatever computation is required to refresh the render details. Notably, it will need to refer to the item class instance (not on heap) which in turn may need to call external managers (eg. SpeedLimitManager). The struct in the task list is then replaced with the refreshed struct.

This means that while the camera isn't moving, the Compute step is sat idle. Additionally, because the layer functionality allows filtered invalidations, we can do micro-computes for individual items (or batches of items) if they change, eg. due to mouse interaction or subtool mode change. In cases of heavy load, we can also spread Compute step across frames.

Code example: Computing a label

The Label property is the class instance created by the Generator step.

            internal bool Refresh(ref OverlayState state) {
                var worldPos = Label.WorldPos;
                
                float distance = (worldPos - state.CameraPos).magnitude;

                if (distance > MAX_CAMERA_DISTANCE || !worldPos.IsOnScreen(out var screenPos)) {
                    //Hide(ref state);
                    return false;
                }

                ScreenPos = screenPos;

                float zoom = 1.0f / distance * 150f;
                float fontSize = Label.TextSize * zoom;

                fontSize = Mathf.Floor(fontSize / FONT_STEP_SIZE) * FONT_STEP_SIZE;

                style_.fontSize = (int)Mathf.Clamp(fontSize, FONT_MIN_SIZE, FONT_MAX_SIZE);
                style_.normal.textColor = Label.TextColor;

                text_ = Label.GetText(IsHovered, ref state);
                Vector2 size = style_.CalcSize(new GUIContent(text_));
                Bounds = new(screenPos.x - (size.x / 2f), screenPos.y, size.x, size.y);

                IsDirty = false;
                Altered = true;
                return true;
            }

Render

This is always processing per frame, but is as fast as possible I think. It iterates the tasks (structs - on heap) and does the GUI render call based entirely on the cached values from Compute stage.

Code example: Rendering a label

internal void Render() => GUI.Label(Bounds, text_, style_);

[krzychu124]

Interface methods/properties must be public and also implementations because interface is a contract. You can set interface as internal but then you have to use explicit implementation (Instead of Custom1 it needs to be ILabelOverlay.Custom1)
Main question is if you really need an interface there?

[originalfoo]

Originally I had delegates for event handlers, etc., but that was a lot of mess (eg. null checks) so the interface and abstract base really cleaned that stuff up.

Have you designed a diagram or something that could describe how you see it or how would it work? I've got a lot of snippets and not implemented ideas worth trying which could fit best

Not yet. I'm playing with caching the acutal overlays, in addition to the usual stuff.

So the code to display node labels looks like:

        public void Render(OverlayRenderSettings settings) {

            // todo: iterate visible nodes form cache manager instead
            for (int nodeId = 1; nodeId < NetManager.MAX_NODE_COUNT; ++nodeId) {
                // validity check done by cache manager
                LabelLayer.Instance.Add(new NodeLabel(nodeId));
            }
        }

NodeLabel class (just playing round with stuff atm):

        internal class NodeLabel : LabelBase, ILabel {

            private const int TEXT_SIZE = 15;
            private readonly Color TEXT_COLOR = new(0f, 0f, 1f);

            public NodeLabel(int nodeId)
                : base(nodeId)
            {
                Text = $"N|{nodeId}";
                TextSize = TEXT_SIZE;
            }

            public override Overlays Overlay => Overlays.Nodes;

            public override OverlayTargets Target => OverlayTargets.Nodes;

            public override Color TextColor => TEXT_COLOR;

            [Hot("Not per-frame, but occasionally called in large batches")]
            public override Vector3 GetWorldPos() =>
                ((ushort)TargetId).ToNode().m_position;

            [Cold("Mouse interaction")]
            public override bool IsInteractive() =>
                OverlayManager.Instance.IsInteractive(Overlay);

            [Cold("Mouse interaction")]
            public override void OnHover(bool mouseInside) {
                if (mouseInside) {
                    // todo: ask manager to throb node
                    TextSize += 3;
                } else {
                    // todo: tell manager to stop node throb
                    TextSize -= 3;
                }
                return;
            }

            [Cold("Mouse interaction")]
            public override bool OnClick(bool mouseDown, bool mouseInside) {
                // todo: tell manager node is selected
                // todo: display additional node data?

                return true; // consumed
            }
        }

Rendering goes something like this (still all WIP ofc):

        [Hot("Every frame while overlays active")]
        internal void Render(OverlayRenderSettings settings, OverlayRenderData data) {
            if (cachedLabels.Count == 0)
                return;

            for (int idx = 0; idx < cachedLabels.Count; idx++) {
                var cache = cachedLabels[idx];

                // if label is dirty, recalcuate render details and save
                if (cache.IsDirty) {

                    var label = cache.Label;

                    // get worldpos
                    cache.WorldPos = label.GetWorldPos();

                    // work out distance from camera
                    float distance = (cache.WorldPos - data.CameraPos).magnitude;

                    // skip if too far away
                    // what if distance from mouse? would need clean way to check that
                    // maybe two iterators - one for camera range, the other mouse range?
                    if (distance > MAX_CAMERA_DISTANCE) {
                        cache.IsHidden = true;
                        cache.IsDirty = false;
                        continue;
                    }

                    // now check if it's on screen
                    if (!cache.WorldPos.WorldToScreenPoint(out var screenPos)) {
                        cache.IsHidden = true;
                        cache.IsDirty = false;
                        continue;
                    }

                    // determine font size based on distance from camera.
                    float zoom = 1.0f / distance * 150f;
                    float fontSize = label.TextSize * zoom;

                    // snap font to nearest 3px step, to font size texture atlases
                    fontSize = Mathf.Floor(fontSize / FONT_STEP_SIZE) * FONT_STEP_SIZE;

                    // apply style, constraining font to reasonable bounds
                    cache.Style.fontSize = (int)Mathf.Clamp(fontSize, FONT_MIN_SIZE, FONT_MAX_SIZE);
                    cache.Style.normal.textColor = label.TextColor;

                    // determine label bounds rect (this is likely expensive - can we cache and scale?)
                    cache.Text = label.Text;
                    Vector2 size = cache.Style.CalcSize(new GUIContent(cache.Text));
                    cache.Bounds = new(screenPos.x - (size.x / 2f), screenPos.y, size.x, size.y);

                    cache.IsDirty = false;
                }

                if (!cache.IsHidden) {
                    // render
                    GUI.Label(cache.Bounds, cache.Text, cache.Style);
                }

                // check if label hovered
                bool hovered = cache.Bounds.Contains(data.MousePos);

                if (hovered) {
                    // handle mouse ineraction if neccessary (infrequent)
                    if (cache.Label.IsInteractive()) {
                        cache.IsHovered = hovered;
                        cache.IsClicked = data.Clicked;
                    }
                } else if (cache.IsHovered) {
                    // let label know it's no longer hovered
                    cache.IsHovered = false;
                    cache.IsClicked = false;
                }
            }
        }

Oh, I also added [Hot] and [Cold] attribs to help illustrate the code. Might add a [Spike] attrib too (indicate where large batch of stuff might happen causing lag spike).

Which, while camera isn't moving is essentially just doing:

// per step of the for loop:

var cache = cachedLabels[idx];

if (cache.IsDirty) {
   // it's not, unless camera moved
}

if (!cache.IsHidden) {
    GUI.Label(cache.Bounds, cache.Text, cache.Style);
}

if (hovered) {
   // mouse stuff
} else if (cache.IsHovered) {
  // will be at most once for a non-hovered item
}

It means that while camera is still, rendering overhead is just checking few bools and a GUI draw if necessary per overlay item.

Note that there's layers now - ground (eg. lane connetors - not even started on that stuff yet), icons (eg. speed limit icons), labels (like node ids, etc).

[krzychu124]

Question: why it is processing everything? 🤔 At first glance, besides rewriting things it will be pretty much as slow as it is currently😕
I'll send you gist how I render things in PF Sandbox, it's not perfect but many times faster than current solution without optimizations

[originalfoo]

At the moment it's because I just haven't got round to thinking about that part yet so just splatted down basic code.

BTW, there's a layer above all the stuff above which essentially detaches overlays from TrafficManagerTool (interactive-mode overlays will still communicate with subtools obviously). But the goal is that we'll be able to use overlays anywhere.

For example - other tools:

        private static readonly ReadOnlyCollection<OverlayRenderSettings> Settings = new(new List<OverlayRenderSettings>() {

            new OverlayRenderSettings {
                Context =
                    OverlayContext.Tool,
                Tool =
                    typeof(NetTool),
                Culling =
                    OverlayCulling.Mouse,
                Interactive =
                    Overlays.None,
                Persistent =
                    Overlays.GroupAwareness,
                Filter =
                    RestrictedVehicles.All,
            },

            new OverlayRenderSettings {
                Context =
                    OverlayContext.Tool,
                Tool =
                    typeof(BulldozeTool),
                Culling =
                    OverlayCulling.Mouse,
                Interactive =
                    Overlays.None,
                Persistent =
                    Overlays.GroupAwareness,
                Filter =
                    RestrictedVehicles.All,
            },

// ...

And info views:

        private static readonly ReadOnlyCollection<OverlayRenderSettings> Settings = new(new List<OverlayRenderSettings>() {
            // Public Transport
            // https://github.com/CitiesSkylinesMods/TMPE/wiki/Public-Transport-Info-View
            new OverlayRenderSettings {
                Context =
                    OverlayContext.Info,
                Info =
                    InfoMode.Transport,
                Culling =
                    OverlayCulling.Mouse,
                Interactive =
                    Overlays.None,
                Persistent =
                    Overlays.GroupTransport,
                Filter =
                    RestrictedVehicles.Transport |
                    RestrictedVehicles.Taxis |
                    RestrictedVehicles.Cars,
            },

            // Natural Resources
            new OverlayRenderSettings {
                Context =
                    OverlayContext.Info,
                Info =
                    InfoMode.NaturalResources,
                Culling =
                    OverlayCulling.Mouse,
                Interactive =
                    Overlays.None,
                Persistent =
                    Overlays.GroupCargo,
                Filter =
                    RestrictedVehicles.Cargo,
            },

// ....

If no "custom" overlay (eg. TMPE subtool is interactive), the tool/infoview overlays kick in automatically - event driven (well, I've not worked out how to detect tool change yet).

And other mods can turn on overlays (like TMPE subtools, they'll be treated as "custom" overlay mode), although it's really complex:

var overlays = OverlaysManager.Instance;

// display stanard situational-awareness overlays
overlays.TurnOn();

// display specific overlays
overlays.TurnOn(
   Overlays.VehicleRestrictions | Overlays.SpeedLimits,
   OverlayCulling.Mouse);

// turn off
overlays.TurnOff();

I'll send you gist how I render things in PF Sandbox, it's not perfect but many times faster than current solution without optimizations

Yes, please do!

[krzychu124]

I've tested both versions and they are pretty much the fastest possible options for collecting all visible nodes and segments.
The only important thing to keep in mind and which may require specific handling is the location when it should be triggered.

https://gist.github.com/krzychu124/9d9208652390c877c7b85d1d17f9679c

Basically my idea was to create general overlay manager which could handle subscriptions and run render methods in the correct timing.
To be able to render things at any time regardless of active tool I would just create new SimulationManager implementing just IRenderableManager.
Like you've mentioned the tricky part will be deciding when it should be visible.

Anyways, above loop may not be useful directly for rendering OnGUI stuff, but might be perfect for preparing which nodes and segments are visible (which I used) since OnGUI is called after everything else. Up to you how you'll handle everything, it could be RenderAll(), Render(single nodeId). Render(nodeId, struct_with_8_segmentIds) etc.

[originalfoo]

The only important thing to keep in mind and which may require specific handling is the location when it should be triggered.

I've got some rough code to do that; basically there's a bunch of criteria (eg. camera move, etc) so I only need to build cache when that happens = much less work.

Are there any names for these magic numbers below? I'm guessing it's dimensions of the render grid or something? Are those dependent on CO code and if so should we pull the values from somewhere during instantiation?

            if (renderGroup.m_instanceMask != 0) {
                int minX = renderGroup.m_x * 270 / 45;
                int minZ = renderGroup.m_z * 270 / 45;
                int maxX = (renderGroup.m_x + 1) * 270 / 45 - 1;
                int maxZ = (renderGroup.m_z + 1) * 270 / 45 - 1;

Also, I assume there's similar grid stuff for buildings, citizens, vehicles, etc? If you have any of that to hand (in case there are subtle differences in code) it would be much appreciated.

I've got a mapping (just rough draft) between overlays and what their requirements are, so at the start of an overlay session I can compile a list of what the "in view" cacheable things are via flags and thus ensure only required things are getting cached.

        private static readonly Dictionary<Overlays, OverlayTargets> Targets = new(IndividualOverlays.Count) {
            { Overlays.PrioritySigns, OverlayTargets.Nodes },
            { Overlays.TrafficLights, OverlayTargets.Nodes },
            { Overlays.SpeedLimits, OverlayTargets.Segments },
            { Overlays.VehicleRestrictions, OverlayTargets.Segments },
            { Overlays.ParkingRestrictions, OverlayTargets.Segments },
            { Overlays.ParkingSpaces, OverlayTargets.Buildings },
            { Overlays.JunctionRestrictions, OverlayTargets.Nodes },
            { Overlays.LaneConnectors, OverlayTargets.Nodes },
            { Overlays.LaneArrows, OverlayTargets.Nodes },
            { Overlays.Nodes, OverlayTargets.Nodes },
            { Overlays.Lanes, OverlayTargets.Lanes },
            { Overlays.Vehicles, OverlayTargets.Vehicles },
            { Overlays.PathUnits, OverlayTargets.None },
            { Overlays.Citizens, OverlayTargets.Citizens },
            { Overlays.Buildings, OverlayTargets.Buildings },
        };

Probably need to rename OverlayTargets to CacheTargets to make intent clearer. Also will need to do some other stuff, eg. some overlays need to know where middle of segment is so that could be done and cached when bulding segments cache (only if there is an overlay that needs it).

[krzychu124]

Are there any names for these magic numbers below? I'm guessing it's dimensions of the render grid or something? Are those dependent on CO code and if so should we pull the values from somewhere during instantiation?

Yeah it's mix from render manager and other managers that render things. Yes it calculates boundaries of grid (minXY/maxXY), sadly nothing to cache but pretty cheap to calculate so shouldn't be an issue here.

Anyways, try to not overcomplicate things just for sake of "future uses". Personally I would focus on separating things(this only does rendering, thing only grouping/caching, thing only creating thing A or B) and implementing it in a way it could be extended but not necessarily now because it may take ages until we see anything usable if we focus on providing support for things that will be added in the unknown future😂

[originalfoo]

See OP :)

[originalfoo]

Given two list of cell indexes, is there some super-fast way to work out what's changed? Specifically to get list(s) of added vs. removed cell ids. And maybe have some cut-off where we say 'rather than doing partial refresh, it's going to be easier to just regenerate' (likely via caching camera position from last refresh and comparing to cusrrent camera position?)

EDIT: For example, I move camera to the left, but half the old screen is still in view. So knowing what the new cells are I can scan just those nodes/segment/etc grids. And, additionally, I can remove nodes/segments/etc from the old list.

Instead of storing just id, store a simple struct eg:

private struct MapEntity {
    internal int Id;
    internal int Cell;
    internal CacheTarget Target;
}

That struct would become the default way of identifying what overlay icons/labels/etc relate to, so it would be very easy to add/remove stuff from layers given list of "cells to remove" and "cells to add".

[originalfoo]

Given two list of cell indexes, is there some super-fast way to work out what's changed?

Just realised there's an easy way to do deltas betweeen grid cells...

// generateCells = stuff from previous map cache invocation
var HashSet<int> prevCells = new(generateCells.m_buffer);
var HashSet<int> repositionCells = new(generateCells.m_size);

generateCells.Clear(); // clear old stuff

for // ...
    for // ...
        var cell = j * 270 + k;

        if (prevCells.Contains(cell)) {
            // layers need to reposition existing task
            repositionCells.Add(cell);
        } else {
            // send to generator to create new layer tasks
            generateCells.Add(cell)
        }
    }
}

I can probably get some extra performance by better management of prevCells hashset (.Clear(), .EnsureCapacity()) between invocations.

When repositioning cells I also know what to remove - anything not in the repositionCells hashset is no longer in camera view! I just need to make sure the layers receive the repositionCells before generators start sending them tasks from generateCells. The main overlay manager can handle that easily.

This also means that if camera is just zooming in and out there will be very little work to do.

[originalfoo]

Hrm, actually, what's in a cell depends on camera position so the above won't work.

But it will work for node id's etc. Will require few extra hashsets but should reliably give me deltas between updates. Also means I don't need to store cell id in the node/segment/etc tables, I can just use uint ids. Will probably use InstanceID as that will make it super-fast to filter lists.

[originalfoo]

@krzychu124 How are you triggering your EndRenderingImpl method? Is it part of class that registers with RenderManager (SimulationManager?), an OnLateUpdate or a harmony postfix?

[krzychu124]

I've created a class which only job is to register itself as a new SimulationManager and loop through registered instances (when creating object I just call pfSim.instance.Register(this), where the class(this) implements interface with render methods, depends on needs). Of course that class unregister itself from managers when removed and "renderables" too.

https://gist.github.com/krzychu124/175843ff6b7479d0f8658131b2733cf3