Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ jobs:
fail_ci_if_error: false

docs:
needs: test
name: Documentation
runs-on: ubuntu-latest
continue-on-error: true
Expand All @@ -75,7 +74,7 @@ jobs:
- name: Setup julia
uses: julia-actions/setup-julia@v2
with:
version: 'lts'
version: '1.12'
arch: x64

- name: Build Docs
Expand Down
2 changes: 0 additions & 2 deletions docs/Project.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
[deps]
Cairo = "159f3aea-2a34-519c-b102-8c37f9878175"
Compose = "a81c6b42-2e10-5240-aca2-a61377ecd94b"
DistributedFactorGraphs = "b5cc3c7e-6572-11e9-2517-99fb8daf2f04"
Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
GraphMakie = "1ecd5474-83a3-4783-bb4f-06765db800d2"
Expand Down
7 changes: 1 addition & 6 deletions docs/src/DataStructure.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,16 +37,11 @@ Accessible properties for each of the factor structures:
- [`VariableSummary`](@ref)
- [`FactorSummary`](@ref)

## DFG Portable and Storeable types
## DFG full types (solvable, portable and storeable)

- [`VariableDFG`](@ref)
- [`FactorDFG`](@ref)

## DFG Full solvable types

- [`VariableCompute`](@ref)
- [`FactorCompute`](@ref)

## Additional Offloaded Data

Additional, larger data can be associated with variables and factors using keyed blob entries.
83 changes: 22 additions & 61 deletions docs/src/GraphData.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,9 @@ Each variable or factor can have a timestamp associated with it.

Tags are a set of symbols that contain identifiers for the variable or factor.

- [`getTags`](@ref)
- [`listTags`](@ref)
- [`mergeTags!`](@ref)
- [`removeTags!`](@ref)
- [`deleteTags!`](@ref)
- [`emptyTags!`](@ref)


Expand All @@ -72,52 +72,11 @@ The solvable flag indicates whether the solver should make use of the variable o

#### Variable Type

The `variableType` is the underlying inference variable type, such as a Pose2.
The `AbstractStateType` is the underlying inference variable type, such as a Pose2.

- [`getVariableType`](@ref)
- [`getStateKind`](@ref)


#### Packed Parametric Estimates

Solved graphs contain packed parametric estimates for the variables, which are keyed by the solution (the default is saved as :default).

For each PPE structure, there are accessors for getting individual values:

- [`getPPEMax`](@ref)
- [`getPPEMean`](@ref)
- [`getPPESuggested`](@ref)


Related functions for getting, adding/updating, and deleting PPE structures:


- [`listPPEs`](@ref)
- [`getPPE`](@ref)
- [`addPPE!`](@ref)
- [`updatePPE!`](@ref)
- [`deletePPE!`](@ref)
- [`mergePPEs!`](@ref)


Example of PPE operations:

```julia
# Add a new PPE of type MeanMaxPPE to :x0
ppe = MeanMaxPPE(:default, [0.0], [0.0], [0.0])
addPPE!(dfg, :x0, ppe)
@show listPPEs(dfg, :x0)
# Get the data back - note that this is a reference to above.
v = getPPE(dfg, :x0, :default)
# Delete it
deletePPE!(dfg, :x0, :default)
# Update add it
updatePPE!(dfg, :x0, ppe, :default)
# Update update it
updatePPE!(dfg, :x0, ppe, :default)
# Bulk copy PPE's for x0 and x1
updatePPE!(dfg, [x0], :default)
```

#### Variable States (Solver Data)

Variable `State`s are used by the IncrementalInference/RoME/Caesar solver.
Expand Down Expand Up @@ -147,23 +106,23 @@ stateBack = getState(dfg, :x0, :parametric)
deleteState!(dfg, :x0, :parametric)
```

#### Metadata
#### Bloblets

Metadata (small data) allows you to assign a dictionary to variables. It is a useful way to
keep small amounts of primative (Strings, Integers, Floats, Bool) data in a variable. As it is stored in the graph
Bloblets allow you to assign a dictionary of key-value [Symbol-String] pairs to nodes. It is a useful way to
keep small amounts of primitive (Strings, Integers, Floats, Bool) data that is stored as a string in a node. As it is stored in the graph
itself, large entries will slow the graph down, so if data should exceed a
few bytes/kb, it should rather be saved in Blobs.


- [`getMetadata`](@ref)
- [`setMetadata!`](@ref)
- [`getVariableBloblet`](@ref)
- [`addVariableBloblet!`](@ref)


Example:

```julia
setMetadata!(x0, Dict("entry"=>"entry value"))
getMetadata(x0)
addVariableBloblet!(dfg, :x0, Bloblet(:bloblet_label,"bloblet value"))
getVariableBloblet(dfg, :x0)
```

#### Big Data
Expand All @@ -179,23 +138,25 @@ you are working with an in-memory graph, the DFG structure contains the graph it

Graphs reside inside a hierarchy made up in the following way:
- Agent
- Metadata
- Bloblets
- Blobentries
- Graph
- Metadata
- Bloblets
- Blobentries

This data can be retrieved with the follow functions:

- [`getAgentMetadata`](@ref)
- [`getGraphMetadata`](@ref)
Agent and Graph bloblets are useful for storing data that is related to the entire graph, and support CRUD operations:

- [`getAgentBloblet`](@ref)
- [`getGraphBloblet`](@ref)

It can be set using the following functions:
- [`addAgentBloblet!`](@ref)
- [`addGraphBloblet!`](@ref)

- [`setAgentMetadata!`](@ref)
- [`setGraphMetadata!`](@ref)
- [`mergeAgentBloblet!`](@ref)
- [`mergeGraphBloblet!`](@ref)

- [`deleteAgentBloblet!`](@ref)
- [`deleteGraphBloblet!`](@ref)

Example of using graph-level data:

Expand Down
12 changes: 12 additions & 0 deletions docs/src/services_ref.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,18 @@ Modules = [DistributedFactorGraphs]
Pages = ["services/CommonAccessors.jl"]
```

## Common
```@autodocs
Modules = [DistributedFactorGraphs]
Pages = [
"services/list.jl",
"services/find.jl",
"services/Tags.jl",
"entities/Bloblet.jl",
"services/Bloblet.jl",
]
```

## DFG Variable Accessors CRUD and SET opperations

```@autodocs
Expand Down
2 changes: 1 addition & 1 deletion src/DataBlobs/services/BlobEntry.jl
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ function gatherBlobentries(
labelFilter = variableLabelFilter,
)
return map(vls) do vl
return vl => getBlobentries(dfg, vl; labelFilter, blobIdFilter)
return vl => getVariableBlobentries(dfg, vl; labelFilter, blobIdFilter)
end
end
const collectBlobentries = gatherBlobentries
Expand Down
8 changes: 4 additions & 4 deletions src/DataBlobs/services/BlobWrappers.jl
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ function loadBlob_Variable(
# hashfunction = sha256,
# checkhash::Bool = true,
)
entry = getBlobentry(dfg, variable_label, entry_label)
entry = getVariableBlobentry(dfg, variable_label, entry_label)
blob = getBlob(dfg, entry)
# checkhash && assertHash(de, db; hashfunction)
return entry, blob
Expand All @@ -82,7 +82,7 @@ function saveBlob_Variable!(
blob::Vector{UInt8},
entry::Blobentry,
)
addBlobentry!(dfg, variable_label, entry)
addVariableBlobentry!(dfg, variable_label, entry)
addBlob!(dfg, entry, blob)
return entry
end
Expand All @@ -100,8 +100,8 @@ function saveBlob_Variable!(
end

function deleteBlob_Variable!(dfg::AbstractDFG, variable_label::Symbol, entry_label::Symbol)
entry = getBlobentry(dfg, variable_label, entry_label)
deleteBlobentry!(dfg, variable_label, entry_label)
entry = getVariableBlobentry(dfg, variable_label, entry_label)
deleteVariableBlobentry!(dfg, variable_label, entry_label)
deleteBlob!(dfg, entry)
return 2
end
Expand Down
27 changes: 0 additions & 27 deletions src/Deprecated.jl
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,6 @@
export FactorCompute
const FactorCompute = FactorDFG

"""
Types valid for small data.
"""
const MetadataTypes = Union{
Int,
Float64,
Expand Down Expand Up @@ -135,18 +132,10 @@ function getVariableTypeName(v::VariableSummary)
return v.statetype
end

"""
$(SIGNATURES)
Get the Metadata entry at `key` for variable `label` in `dfg`
"""
function getMetadata(dfg::AbstractDFG, label::Symbol, key::Symbol)
return getVariable(dfg, label).smallData[key]
end

"""
$(SIGNATURES)
Add a Metadata pair `key=>value` for variable `label` in `dfg`
"""
function addMetadata!(dfg::AbstractDFG, label::Symbol, pair::Pair{Symbol, <:MetadataTypes})
v = getVariable(dfg, label)
haskey(v.smallData, pair.first) && throw(LabelExistsError("Metadata", pair.first))
Expand All @@ -155,10 +144,6 @@ function addMetadata!(dfg::AbstractDFG, label::Symbol, pair::Pair{Symbol, <:Meta
return v.smallData #or pair TODO
end

"""
$(SIGNATURES)
Update a Metadata pair `key=>value` for variable `label` in `dfg`
"""
function updateMetadata!(
dfg::AbstractDFG,
label::Symbol,
Expand All @@ -174,30 +159,18 @@ function updateMetadata!(
return v.smallData #or pair TODO
end

"""
$(SIGNATURES)
Delete a Metadata entry at `key` for variable `label` in `dfg`
"""
function deleteMetadata!(dfg::AbstractDFG, label::Symbol, key::Symbol)
v = getVariable(dfg, label)
pop!(v.smallData, key)
mergeVariable!(dfg, v)
return 1
end

"""
$(SIGNATURES)
List all Metadata keys for a variable `label` in `dfg`
"""
function listMetadata(dfg::AbstractDFG, label::Symbol)
v = getVariable(dfg, label)
return collect(keys(v.smallData)) #or pair TODO
end

"""
$(SIGNATURES)
Empty all Metadata from variable `label` in `dfg`
"""
function emptyMetadata!(dfg::AbstractDFG, label::Symbol)
v = getVariable(dfg, label)
empty!(v.smallData)
Expand Down
12 changes: 10 additions & 2 deletions src/DistributedFactorGraphs.jl
Original file line number Diff line number Diff line change
Expand Up @@ -303,8 +303,16 @@ export deleteVariableBloblet!
export listVariableBloblets

export getAgentBloblet
export getGraphBloblet
export addAgentBloblet!
export mergeAgentBloblet!
export deleteAgentBloblet!
export listAgentBloblets

export getGraphBloblet
export addGraphBloblet!
export mergeGraphBloblet!
export deleteGraphBloblet!
export listGraphBloblets
##------------------------------------------------------------------------------
## FileDFG
##------------------------------------------------------------------------------
Expand Down Expand Up @@ -451,13 +459,13 @@ const unstable_functions::Vector{Symbol} = [
:getVariableTypeName,
:getStateKind,
:setTimestamp,
:setMetadata!, # no set, use add merge
:setAgentMetadata!,
:setGraphMetadata!,
# :getSolverDataDict,# obsolete
:getAddHistory,

#Deprecated in v0.28
:setMetadata!, # no set, use add merge
:AbstractRelativeMinimize,
:AbstractManifoldMinimize,
:AbstractPrior,
Expand Down
16 changes: 9 additions & 7 deletions src/FileDFG/services/FileDFG.jl
Original file line number Diff line number Diff line change
Expand Up @@ -39,18 +39,18 @@ function saveDFG(folder::AbstractString, dfg::AbstractDFG)
map(f -> rm("$varFolder/$f"), readdir(varFolder))
map(f -> rm("$factorFolder/$f"), readdir(factorFolder))
# Variables
@showprogress "saving variables" for v in variables
@showprogress desc = "saving variables" for v in variables
# vPacked = packVariable(v)
JSON.json("$varFolder/$(v.label).json", v; style = DFGJSONStyle())
end
# Factors
@showprogress "saving factors" for f in factors
@showprogress desc = "saving factors" for f in factors
JSON.json("$factorFolder/$(f.label).json", f; style = DFGJSONStyle())
end

#GraphsDFG nodes
@assert isa(dfg, GraphsDFG) "only metadata for GraphsDFG are supported"
p = Progress(4, "Saving DFG Nodes")
p = Progress(4; desc = "Saving DFG Nodes")
JSON.json("$savepath/graphroot.json", dfg.graph; style = DFGJSONStyle())
next!(p)
JSON.json("$savepath/agent.json", dfg.agent; style = DFGJSONStyle())
Expand Down Expand Up @@ -119,7 +119,9 @@ function loadDFG!(
variablefiles = readdir(joinpath(loaddir, "variables"); sort = false, join = true)

# type instability on `variables` as either `::Vector{Variable}` or `::Vector{VariableCompute{<:}}` (vector of abstract)
variables = @showprogress 1 "loading variables" asyncmap(variablefiles) do file
variables = @showprogress dt=1 desc = "loading variables" asyncmap(
variablefiles,
) do file
v = JSON.parsefile(file, V; style = DFGJSONStyle())
return addVariable!(dfgLoadInto, v)
end
Expand All @@ -128,7 +130,7 @@ function loadDFG!(

factorfiles = readdir(joinpath(loaddir, "factors"); sort = false, join = true)

factors = @showprogress 1 "loading factors" asyncmap(factorfiles) do file
factors = @showprogress dt=1 desc = "loading factors" asyncmap(factorfiles) do file
f = JSON.parsefile(file, F; style = DFGJSONStyle())
return addFactor!(dfgLoadInto, f)
end
Expand All @@ -137,7 +139,7 @@ function loadDFG!(

if isa(dfgLoadInto, GraphsDFG) && getTypeDFGFactors(dfgLoadInto) <: FactorDFG
# Finally, rebuild the CCW's for the factors to completely reinflate them
@showprogress 1 "Rebuilding factor solver cache" for factor in factors
@showprogress dt=1 desc = "Rebuilding factor solver cache" for factor in factors
rebuildFactorCache!(dfgLoadInto, factor)
end
end
Expand Down Expand Up @@ -170,7 +172,7 @@ function loadDFG(file::AbstractString)
loaddir = Tar.extract(hdr -> contains(hdr.path, dfgnodenames), tar)
close(tar)

progess = Progress(4, "Loading DFG Nodes")
progess = Progress(4; desc = "Loading DFG Nodes")
agent = JSON.parsefile(joinpath(loaddir, "agent.json"), Agent; style = DFGJSONStyle())
next!(progess)
graph = JSON.parsefile(
Expand Down
Loading
Loading