Data type checking - And a few other things #51
Conversation
Allows multiple outputs to be connected to the same input
This reverts commit eb324d2.
This reverts commit 450a13f.
- Implemented Type Checking for Data types, not payload - Implemented a PayloadData for checking if the payload is the correct type for this Data - Implemented various built-in Data types; number module data types, collections module data types, string, bytes - Made node blue prints into dataclasses
- Added an updated and an output_changed event in the Node class
- Improved Connection validity test. Returns an enum and a message
- Added a simple test in data_objects.py to check for type checking and connection validity
- Ensured that the built_in data types have data.built_in.{node_class_name} identifiers
Added a new class indicating some kind of progress. Utilized the class in Node to provide an API for setting a node's update progress (in case of threading etc inside the node)
- Implemented a way to associate payload to data types - Added more tests -
- Added some additional ConnValidTypes and two new functionsin Flow for connection checking
Fixed bug where out_data_type or inp_data_type were None
Flow.can_nodes_connect() now returns a ConnValidType.INPUT_TAKEN if input is connected to another output. This is needed because graph_adj_rev only handles a single inp -> output
Fixed the order in which the various ConnValidTypes are checkd in Flow.can_nodes_connect()
|
Hey, good work! Thanks for writing this up so clearly. Some initial notes:
|
|
Hello!
I completely forgot to update the post. I noticed that the if check isn't good, and changed it to this (updated in original post as well): def check_valid_data(out_data_type: Type[Data], inp_data_type: Type[Data]) -> bool:
"""
Returns true if input data can accept the output data, otherwise false
None type is treated as the default Data type
"""
if inp_data_type is None:
inp_data_type = Data
if out_data_type is None:
out_data_type = Data
return issubclass(out_data_type, inp_data_type)
The messages were supposed to be like "compile" error messages to be honest. I'll make it return only the enum.
The
Similar to progress_state.mp4This particular example doesn't apply to the original Ryven due to many changes in the application (the node logic runs on a separate thread). However, I can definitely see it being useful in Ryven where a node has to load big data, i.e. from a CSV file.
Will do.
I actually wasn't trying to fix this. I implemented the payload = self.input_payload(1)
# as opposed to
data = self.input(1)
if data is not None:
payload = data.payloadThe I think the |
- Removed 'data.' from built-in types identifiers. - All connection validation returns a ConnValidType instead of (ConnValidType, str). The messages were moved to a `get_error_message` function in `NodePort.py` - Removed the added `updated` event in the nodes - Renamed `Node.outputs` and Node.inputs` to `Node._outputs` and `Node._inputs` - Utilized typing.TYPE_CHECKING to add type hinting to (almost?) all the files that couldn't import the type due to circular imports.
|
The last pushes have the following changes:
I'll be waiting to further discuss the |
might make it a classmethod of
fair, but that's no reason to ruin ryvencore performance, but rather to remove the feature from Ryven. Ryven should mostly adapt to ryvencore, not the other way around.1
...just as they can implement progress bars? ryvencore aims to only provide the basics and a modular architecture to build everything else on top (hence the name); I feel this is a little too specific and the previous argument still stands.
IMO this is a perfect example of something that would add a lot of value in some sort of community-driven node feature ecosystem for Ryven, no? (which doesn't exist yet)
yes, I can spin this thought to ridiculous scenarios where the behavior of a data object essentially depends on things that neither the developer, nor the user actively control due to conflicts. The Footnotes
|
Sure!
I'll remove the implicit conversions, I too think it was a bit too much, especially for a base version of a node system that implements its own As per the An easy solution can be to simply check the def emit(self, *args):
if len(self._slot_priorities) == 0:
return
# notice that dicts are insertion ordered since python 3.6
for nice, cb_set in self._slots.items():
for cb in cb_set:
cb(*args)This however doesn't solve the issue of having instantiated unused sets. A better solution would simply be to instantiate the set when it's needed and reorder the dictionary: class Event:
def __init__(self, *args):
self.args = args
self._slots: Dict[int, Set] = {}
self._slot_priorities = {}
def sub(self, callback, nice=0):
# this can be commented for indefinite number of -nice- callbacks
assert -5 <= nice <= 10
assert self._slot_priorities.get(callback) is None
if self._slots.get(nice) is None:
self._slots[nice] = set()
self._slots = dict(sorted(self._slots.items()))
self._slots[nice].add(callback)
self._slot_priorities[callback] = nice
def unsub(self, callback):
nice = self._slot_priorities[callback]
cb_set = self._slots[nice]
cb_set.remove(callback)
del self._slot_priorities[callback]
# no need to sort when deleting
if len(cb_set) == 0:
del self._slots[nice]
def emit(self, *args):
# notice that dicts are insertion ordered since python 3.6
for nice, cb_set in self._slots.items():
for cb in cb_set:
cb(*args)We can even say that the Per the I'll argue again that an
Even if ryvencore doesn't provide a progress api out of the box, Ryven could've a progress bar implementation. That might justify leaving the progress API out of ryvencore, but still being able to access a GUI implementation in Ryven. Still, I think at least the Could you perhaps see a base node, i.e |
Removed `register_payload_to_data` and `set_output_payload`
|
As an update to the above, here's a refined version of class Event:
def __init__(self, *args):
self.args = args
self._slots: Dict[int, Set] = {}
self._ordered_slot_pos = []
self._slot_priorities = {}
def sub(self, callback, nice=0):
assert -5 <= nice <= 10
assert self._slot_priorities.get(callback) is None
cb_set = self._slots.get(nice)
if cb_set is None:
cb_set = self._slots[nice] = set()
insort(self._ordered_slot_pos, nice)
cb_set.add(callback)
self._slot_priorities[callback] = nice
def unsub(self, callback):
nice = self._slot_priorities[callback]
cb_set = self._slots[nice]
cb_set.remove(callback)
del self._slot_priorities[callback]
if len(cb_set) == 0:
del self._slots[nice]
self._ordered_slot_pos.remove(nice)
def emit(self, *args):
# notice we're using the sorted list to run through the events
for nice in self._ordered_slot_pos:
for cb in self._slots[nice]:
cb(*args) |
Fixed a left-over Nodes.outputs to Nodes._outputs
This PR includes type checking with the
Dataclass when connecting nodes as well as when passing data between ports. It also includes some QoL changes. Specifically...Data and Type Checking
Due to various changes, a new folder / module named data has been introduced.
Type checking is done via a simple function defined in
Data.py:Another function is defined in
NodePort.pyfor connection validity. It uses a new enum type defined inRC.py:Code
As shown above, ports can be given an
allowed_datain the constructor, which is serialized along with the port. Finally, setting an output value asserts for the correct type like this:Along with these, the PR includes a
_BuiltInDatatype inData.pyfor defining various built-in types associated with payload types of the python standard library. Types from a specific parent class can be searched usingget_built_in_data_typesdefined incore.py. The built-in types are automatically registered when aSessionis created, with an identifier ofdata.built_in.{class_name}. The built-int types are:core.py,PayloadData,StringData,BytesDatadata.built_in.collection.abcsub-module, one-to-one related with the standard librariesabccollectionsdata.built_in.collection.data_typessub-module, namelyListData,TupleData,DictData,OrderedDictData,SetData,FrozenSetData,QueueDatadata.built_in.numberssub-module, one-to-one related with thenumbersmodule in the standard library. These areNumberData,ComplexData,RealData,RationalData,IntegerData.The types are made to assert that a payload's type must be a sub-type of a base class. All the classes except the numbers assert that the payload type is correct. The
numberclasses attempt to cast the payload to a correct default type if its type is incorrect.While making this functionality, I noticed that some QoL could be made to improve setting inputs and getting outputs.
payload_to_datadict,data_to_payloaddict,register_payload_to_data-register_payload_to_data_multifunctions are provided inData.pyfor the developer to associate payload types to a certain data type. With them, a virtual class-method namedregister_payload_type(cls)is defined in theDataclass and called when registering the class in aSessionfor inheritance-based payload to data association. All the above built-in types register themselves, either when imported or using the inheritance method.i.e
Using the above, these functions have been implemented in the
Nodeclass.Code
The above functions 'skip' over the need of manually instantiating a data class, provided that dataclass works with a constructor that only needs a payload.
(At this point, I'd like to say that the
input()function might be needing a rename, since it can be confused with theinputslist, despite the fact that the first returns aDatainstance and the second stores the ports)Much of the functionality has been tested in a third
DataTypesBuiltIntest defined indata_objects.pyQuality of Life
Some other improvements include additional events in
Node:updated: invoked when theupdate()function exitsoutput_updated: invoked when an output is updatedprogress_updated: invoked when setting the progress in a nodeRegarding the third point, a
ProgressStateclass has been added inRC.pyfor defining the progress inside a node, for example when using threads or multi-processes. It's a simple class and the associated functions are defined in the PROGRESS section ofNode.Lastly, the
NodePortTypeclasses were made intodataclasses.These are (hopefully :D) all the important changes and additions I've made in this PR.