Add BigData #131
Add BigData #131
Conversation
Codecov Report
@@ Coverage Diff @@
## master #131 +/- ##
==========================================
+ Coverage 49.97% 50.53% +0.56%
==========================================
Files 28 29 +1
Lines 1763 1783 +20
==========================================
+ Hits 881 901 +20
Misses 882 882
Continue to review full report at Codecov.
|
|
Initial idea for big data to get us going |
| solverDataDict::Dict{Symbol, VariableNodeData} | ||
| smallData::Dict{String, String} | ||
| bigData::Any | ||
| bigData::Dict{Symbol, AbstractBigDataEntry} |
There was a problem hiding this comment.
can we use Option 2 method here, i.e.:
bigData::Dict{Symbol, <: AbstractBigDataEntry}There was a problem hiding this comment.
The whole advantage from this approach is to have an abstract container. I did some benchmarks and an abstract dict with composite types in does not seem to perform worse than a typed dict.
I'm more than open for suggestions of other ways to do it, but <: AbstractBigDataEntry will not work in this case.
The advantage is in the flexibility, the types I added for illustration and the ones I currently find useful are:
struct MongodbBigDataEntry <: AbstractBigDataEntry
key::Symbol
oid::NTuple{12, UInt8} #mongodb object id
#maybe other fields such as:
#flags::Bool ready, valid, locked, permissions
#MIMEType::Symbol
end
struct FileBigDataEntry <: AbstractBigDataEntry
key::Symbol
filename::String
end
One can just as easily define other types as long as it has a key field, maybe in memory - stored in an dictionary:
If you don't mind it not working in the cloud.
struct MemoryBigDataEntry <: AbstractBigDataEntry
key::Symbol
dictionaryModule::Symbol
dictionaryReference::Symbol
end
The alternatives i could think of were to standardize BigDataEntry, but I don't know what to put in there and you lose dispatch. Or to just keep the keys in a Set{Symbol} kind of way.
|
We can close this right, it's merged into master already as part of megaPR? |
|
|
|
xref #97 |
Big data added to DFGVariable