refactor assigner

[matt-beanland]

now that calculations and aggregates and calculation expressions are supported in the datalayer, we should improve the assigner.
A good test case is a vlan with 4000 tags. We should exercise and benchmark assigners of 100, 1000, 10000 things ideally.

[matt-beanland]

On the assigner + aggregates/calculations:

The assigner pattern (:assign/:unassign/:auto_assign operations, AssignableValue with :first/:last/:free range) looks like it's doing capacity and availability reasoning that currently lives in application code. Aggregates and expression calculations could push a lot of that into the graph query:

:count of active assignments → capacity check in a single query rather than loading and counting in Elixir
:list of assignments → replace manual traversals
Expression calculations → derive :free slots (max - count) as a calculated field directly on the resource
Worth experimenting in ash_neo4j first to understand what the aggregate pushdown looks like for this kind of range/capacity query, then bring it back.

[matt-beanland]

I think there is 'parent' concept in expression that lets you go up to the instance holding a characteristic and grab a calculation when evaluating an embedded resource. So when we have a characteristic value like assigner we want live, we may be able to just put a calculation or aggregate on the isntance (by dsl transformation), and have it picked up by the assignableValue (now an embedded resource).

No it doesn't work that way. In filter/runtime.ex:372-373:

defp resolve_expr({:_parent, _}, _, _, _, _), do: :unknown
defp resolve_expr({:_parent, _, _}, _, _, _, _), do: :unknown
parent() resolves to :unknown at runtime — it's a data layer construct pushed down to Cypher/SQL, not an Elixir-side evaluation mechanism. Embedded resources evaluate in Elixir (no datastore), so parent() inside an embedded resource calculation won't work.

The architecture needs to flip: instead of the embedded value reaching up to the instance, the instance reaches down to the config. The DSL transformation is still the right mechanism, just the computation stays on the instance:

AssignableValue (TypedStruct or embedded)
→ static config: first, last, algorithm

Instance (graph node)
→ aggregate :assigned_vlan_ids ← injected by DSL transformer
→ calculation :free_vlan_count ← injected by DSL transformer
expr(vlan_id_config.last - vlan_id_config.first + 1 - count(assigned_vlan_ids))
The transformer detects a characteristic declared with an assignable type and injects both the aggregate and the calculation onto the instance. The aggregate IS pushed to Cypher (efficient for 4000 VLAN tags). The calculation is Elixir-side but operating on already-aggregated data — just arithmetic.

So on the instance when we load we

Load the Instance — characteristic attribute comes up from Neo4j with the stored AssignableValue (static config: first, last, algorithm)
The injected aggregate also runs — queries the Assignment nodes and returns the live count or list
A calculation (also injected by the transformer) merges the live result back into the AssignableValue struct, populating free
So free stops being a stored field and becomes a computed one — the stored struct just carries the config, the live view is the merged result. Consumers load the Instance with the calculated characteristic and get back a fully hydrated AssignableValue where free reflects reality.

The calculation module the transformer injects would look roughly like:

def load(_query, _opts, _context), do: {:ok, [:assigned_vlan_ids, :vlan_assignment]}

def calculate(records, _opts, _context) do
Enum.map(records, fn record ->
config = record.vlan_assignment
free = (config.last - config.first + 1) - length(record.assigned_vlan_ids)
{:ok, %{config | free: free}}
end)
end
The aggregate does the heavy work in Cypher; the calculation is pure Elixir arithmetic on the result. The consumer just loads [:live_vlan_assignment] and the struct is hydrated — no awareness of the underlying aggregate needed.

The free field on the stored AssignableValue could even be removed entirely, or kept as a fallback for when the calculation isn't loaded.

This works when we load through the instance. Anything reading the characteristic (say using Provider API) won't see the dynamic values. We will need dsl for this.

[matt-beanland]

Now that #128 (BaseCharacteristic) is planned, this refactor should be sequenced after it. AssignableValue becomes a BaseCharacteristic-derived resource with first, last, algorithm as real Ash attributes — the DSL-injected aggregate and calculation live directly on the characteristic resource, removing the need to solve Ash.Type.Dynamic in expressions entirely. The parent() issue described above goes away because we're no longer working with an embedded type reaching up to the instance.

Depends on: #128

[matt-beanland]

decided that we'd simplify the structure for the sub-resource assignment case, such that we'd synthesise the characteristic on the relationship from relationship properties.

BaseRelationship — new fragment, typed relationships with Ash attributes backed by Neo4j relationship properties
Assigner.AssignmentRelationship — BaseRelationship-derived, ships with the assigner, carries the slot attribute
AssignableCharacteristic — BaseCharacteristic-derived, carries first/last/algorithm/assignable_type; free is an aggregate over AssignmentRelationship edges
Provider DSL — assign in behaviour do wires all three together; transformer injects the action and aggregate; instance namespace stays clean

[matt-beanland]

Goal

Replace imperative Elixir assignment bookkeeping with Ash-native aggregates, typed resources, and a provider DSL declaration. The graph becomes the source of truth; no manual counters or in-memory queries.

Store the assigned value directly on the Relationship node as attributes — pool, thing, assigned — rather than creating a separate Characteristic node just to hold an integer.

Phase 1 — Add assignment attributes to Relationship

Add three attributes to Diffo.Provider.Relationship: pool (atom), thing (atom), assigned (integer). Assignment relationships populate these; the characteristics array is left unused for :assignedTo type. JSON serialiser gets a branch for :assignedTo that renders them appropriately.

Add a second conditional identity permanently to the Relationship resource — no injection required:

identity :unique_assignment, [:source_id, :target_id, :pool, :thing, :assigned] do
where expr(type == :assignedTo)
end
This supports multiple assignments between the same pair (e.g. two ports for redundancy) as long as each is a distinct pool/thing/assigned combination.

Phase 2 — AssignableCharacteristic

Migrate AssignableValue TypedStruct to a BaseCharacteristic-derived resource. Attributes: first, last, assignable_type, algorithm — no free (computed). AssignableCharacteristic.Value TypedStruct for JSON. Assigner accesses fields directly, no unwrapping.

Phase 3 — Update Assigner logic

Replace relate_is_assigned with a single create_relationship carrying pool, thing, assigned instead of creating a Characteristic node first. Replace unrelate_is_assigned with a Relationship lookup by pool/thing/assigned. Drop Characteristic node creation, decrement_free, increment_free.

Phase 4 — Aggregates and free-list calculation

Aggregate on instance counting Relationship records where type == :assignedTo AND pool == pool_name AND thing == thing_name AND source_id == this.id. Calculation: free_count = last - first + 1 - assigned_count. Replace in-memory first..last -- assigned with an expression query.

Prerequisite: confirm AshNeo4j supports this aggregate shape.

Phase 5 — Provider DSL pools section

New pools do section in provider do alongside characteristics and features. Declares pool existence with algorithm and assignable_type defaults — not limits, which are per-instance and runtime-configurable:

pools do
pool :cores, algorithm: :lowest, assignable_type: "tensor"
end
Transformer injects typed pool arguments into build (e.g. cores_first, cores_last) replacing the generic characteristic_value_updates black box. Also injects a resize action for runtime pool changes.

Dependencies

Phases 1, 2, and 3 can proceed together. Phase 4 needs phase 1 done and the AshNeo4j aggregate question answered. Phase 5 needs all of the above.

[matt-beanland]

Phases 1-3 done:

All 713 tests pass. Here's a summary of what changed:

Phase 1 — Diffo.Provider.Relationship:

Added pool, thing, assigned attributes (all optional atoms/integer)
Added create :create_assignment action (sets type: :assignedTo automatically)
Added identity :unique_assignment on [:source_id, :target_id, :pool, :thing, :assigned] where type == :assignedTo
JSON customize now renders a synthetic [%{name: thing, value: assigned}] characteristic list for assignment relationships (only when assigned is non-nil, so old-style relationships are unaffected)

Phase 2 — New AssignableCharacteristic:

BaseCharacteristic-derived resource with first, last, assignable_type, algorithm (no stored free)
AssignableCharacteristic.Value TypedStruct for ordered JSON encoding
Registered in Diffo.Provider domain and the Servo test domain

Phase 3 — Rewritten Assigner:

relate_is_assigned: creates one Relationship with pool/thing/assigned — no Characteristic node
unrelate_is_assigned: queries Relationship by pool/thing/assigned via filter_input, then Ash.destroy
next/assignable?/free_values: query AssignableCharacteristic and current Relationship records from DB — no in-memory traversal, no increment/decrement

[matt-beanland]

checked in:

Phase 4 — Aggregates and free-list calculation

The free_values/5 function is gone — AssignableCharacteristic now carries an assigned_values calculation that fetches the used integers directly, and Assigner computes the free set with Enum.to_list(char.first..char.last) -- char.assigned_values in memory after loading the calculation with Ash.Query.load(assigned_values: [thing: thing]).

[matt-beanland]

Phase 5 - Provider DSL Pools Section

New files:

pool.ex — Pool struct + create_pools/2 (builds AssignableCharacteristic during :build) + update_pools/3 (applies value updates during :define)
persist_pools.ex — bakes pools/0 on the module
verify_pools.ex — rejects duplicate pool names

DSL changes:

pools do / pool :name, :thing / end section added to provider do
pool/1 helper generated on each Instance module by the transformer
Pool.create_pools wired into build_after

Assigner:

New Assigner.assign/3 — looks up thing from result.struct.pool(pool_name); delegates to assign/4
Card test resource updated:

# before: characteristic :ports, AssignableCharacteristic + Assigner.assign(result, changeset, :ports, :port)
# after:
pools do
  pool :ports, :port
end
# ...
Assigner.assign(result, changeset, :ports)

[matt-beanland]

Updated docs and agent guidance. All docs are updated. Here's what changed:

usage-rules.md — new pools do section with full DSL reference, define/assign action examples, updated generated functions list, and three new common-mistakes entries covering pools do, AssignableValue, and assign/3 vs assign/4.

AGENTS.md — added pool.ex, assigner/, and assigned_values.ex to the project structure tree; added pools do block to the Instance DSL example; added four new pool-related agent mistakes.

documentation/how_to/use_diffo_provider_extension.livemd — rewrote the assigner narrative (overview paragraph, pools do in the provider do section list, GPU resource example, define action, assign action, usage calls, Neo4j browser description, "What Next?" summary). The old AssignableValue / Characteristic.update_values / assign/4 pattern is gone throughout.

documentation/dsls/DSL-Diffo.Provider.Extension.md — regenerated via mix spark.cheat_sheets; now includes the full provider.pools and provider.pools.pool sections.

lib/diffo/provider/extension.ex module doc — added pools do / pool :ports, :port / end to the Instance example so the regenerated cheat sheet reflects it.

[matt-beanland]

Phase 6 was to create a BaseRelationship fragment and have AssignerRelationship be an extension. We renamed this to AssignedToRelationship.

Phase 6 — BaseRelationship + AssignedToRelationship

Problem: Diffo.Provider.Relationship was a monolith carrying both TMF service/resource relationship attributes and pool assignment attributes (pool, thing, assigned, :create_assignment action). Assignment records and TMF records were mixed in instance.forward_relationships, requiring callers to filter by type == :assignedTo.

Solution: Split into a shared fragment and two distinct resources.

BaseRelationship — Spark fragment providing common attributes (id, type, target_href, target_type), belongs_to :source/:target, validations, and default actions. Provides AshNeo4j.DataLayer and AshJason.Resource.
AssignedToRelationship — BaseRelationship-derived resource for pool assignments. Neo4j label :AssignedToRelationship (derived from module name, stable under :Provider domain label regardless of calling domain). Carries pool, thing, assigned. Own create_assignment action and unique_assignment identity.
Relationship — refactored to use BaseRelationship. Assignment attributes and :create_assignment action removed. JSON customize no longer has the :assignedTo branch.
BaseInstance — gains has_many :assignments, AssignedToRelationship, matching neo4j relate entry ({:assignments, :RELATES, :outgoing, :AssignedToRelationship}), preparations load, and jason pick.
Instance.Util.relationships/1 — combines forward_relationships ++ assignments for JSON rendering; uses Map.get(rel, :alias) to safely handle structs without :alias.
Domain function create_assignment_relationship renamed to create_assigned_to_relationship. Assigner and AssignedValues calculation updated to query AssignedToRelationship.
Tests updated: card.assignments replaces card.forward_relationships |> filter(type == :assignedTo)

Raised #134 as now we can have Relationship DSL to enforce some structure, and later behaviour, other than the assigner.

[matt-beanland]

Phase 7 — Resource naming: distinct Neo4j labels

Problem: AshNeo4j writes three labels on create — [:Domain, :ResourceName, :FragmentBase] — but uses only the resource label (final module segment) for reads, updates, and destroys. Diffo.Test.Instance.Card and Diffo.Test.Characteristic.Card both derive label :Card, causing query collisions.

Plan:

Rename Diffo.Test.Instance.Card → Diffo.Test.Instance.CardInstance (label :CardInstance)
Rename Diffo.Test.Characteristic.Card → Diffo.Test.Characteristic.CardCharacteristic (label :CardCharacteristic)
Update all references — Servo domain, assigner_test.exs, characteristic test, card.ex aliases
Audit remaining test resources (Shelf, others) for the same collision risk
Document the convention in usage-rules.md and AGENTS.md: when a BaseInstance-derived resource shares a domain concept name with a BaseCharacteristic-derived resource, the final module segment must differ — natural convention is Instance / Characteristic

[matt-beanland]

raised diffo-dev/ash_neo4j#257 to improve ash_neo4j so we don't need duck-typing which is ugly and error prone.