Replies: 3 comments 1 reply
-
Hi @rciric! Thanks for the questions. Wow, I am beyond impressed. This is a challenging part of the codebase. What you want to do makes sense to me, and I suspect you're right that it would give good performance for UDTs of this size. Using UDTs like this would be the "GraphBLAS way" to do things IMHO. I added UDTs to Thanks so much for the gist ❤️ . Looks great and surprisingly readable at a glance (beauty is in the eye of the beholder ;) ). I will take a closer look at your code ASAP, and then we can decide how we can get the functionality into In my final comment #177 (comment), I call out the possibility of default operators to "just work" on array UDTs, such as Feel free to share any more details in this thread that you think may be useful. I'll read it and try to use it. Do you know of any shortcomings to your current solution? Are you blocked by anything, and how urgent are things to get unblocked or improved? We try to be responsive to user needs.
Sorry about that! We try to give a deprecation cycle of at least six months for breaking changes if possible. Moving things to Takeaways and actions
Thanks again @rciric, and welcome to our community 🎉 |
Beta Was this translation helpful? Give feedback.
-
To clarify, this is an example workflow that I want to "just work" in >>> import graphblas as gb
>>> A = gb.Matrix("int[3]", 2, 2)
>>> A << (1, 2, 3)
>>> A
"M_0" nvals nrows ncols dtype format
gb.Matrix 4 2 2 INT64[3] fullr (iso)
-----------------------------------------------------
0 1
0 [1, 2, 3] [1, 2, 3]
1 [1, 2, 3] [1, 2, 3]
>>> B = gb.Matrix("int[3]", 2, 2)
>>> B << (2, 3, 4)
>>> B
"M_1" nvals nrows ncols dtype format
gb.Matrix 4 2 2 INT64[3] fullr (iso)
-----------------------------------------------------
0 1
0 [2, 3, 4] [2, 3, 4]
1 [2, 3, 4] [2, 3, 4]
>>> (A @ B).new()
"M_2" nvals nrows ncols dtype format
gb.Matrix 4 2 2 INT64[3] fullr (iso)
-----------------------------------------------------
0 1
0 [4, 12, 24] [4, 12, 24]
1 [4, 12, 24] [4, 12, 24] The final expression will be the most difficult to do, but this is my target end-state. The earlier expressions need these PRs-- #299 and #300 --which make using UDTs extra-nice imho. |
Beta Was this translation helpful? Give feedback.
-
Hello, and thanks for being so responsive! It’s great to hear you’ve planned on supporting this functionality — I would definitely be happy to work with you all to get a solution (particularly for the last case above) into the code base. I can also open issues for some of the specific troublespots I mentioned. There are a couple of other projects I’m working on simultaneously, so I will try to work with you all toward a preliminary PR in the next couple weeks, but no promises yet IMO the biggest deficiencies in my implementation, relative to the above and your #177 comment, are (1) that I’ve only hard-coded solutions for one elementary operator of each type, and (2) that the implementation defines and registers all new ops as part of the creation of the vector/array UDT, which is done explicitly. I’ll say a little bit about each of these issues below, and my working model of how they might be fixed.
Or, perhaps there is a better way to do this (e.g., that wouldn't require a separate JIT for each new array shape)? Preliminarily, I suspect another source of difficulty integrating my changes would likely stem from reconciling some of the patches with existing code. Off the top of my head, there are the patched op classes, There’s definitely no urgency/blocking here — there are plenty of other features I’m also experimenting with for my use case, etc. (To summarise briefly without going off-topic, the application is something DL-adjacent but not quite DL — I can share more details at the meeting / on the chat, and maybe we can see whether / how it might feasibly fit into the GraphBLAS ecosystem.) Thanks again, and happy to contribute however I can! |
Beta Was this translation helpful? Give feedback.
-
Hello all, and thanks for the incredible work.
The problem
I have several use cases for GraphBLAS that involve either multi-channel graphs or batched graphs, where the sparsity pattern is the same across a vector of observations. For instance, this occurs when we have multiple data channels associated with each edge in a fixed mesh or grid, or when we have a batch of graphs whose connectivity structure is identical but whose weights are different.
I’m not sure if there’s an “out-of-the-box” or recommended way of using
python-graphblas
for this class of problem, but the most direct approach (expectedly) fails with an error like'plus_times does not work with FP32[100]'
(for batch dim 100 and float32 datatype). To address these use cases, I’m thus using the UDT utilities ofpython-graphblas
to create vector datatypes and define operators over those types. The SuiteSparse user guide seems to advise against this (the bit aboutgeneric
kernels in Performance, section 14.2), but I’ve found we still get substantially improved performance relative to the most naive loop this way, provided that we can amortise the compilation cost (i.e., provided that the “batch” or “channel” dimension remains the same across subsequent calls).The current solution (not great)
Directly registering new operators with the
is_udt=True
flag fails for my case, for several reasons related to C pointer datatypes. (I can provide more details if this problem is of interest to others. There are some basic notes in the linked gist below, but it's not a pretty piece of code.)I’ve thus created some very hackish patches of
_get_udt_wrapper
and the_compile_udt
methods of operator classes that allow the vectorisation to work in principle (at least for the small subset of cases I’ve tested). But these implementations are very brittle — they’re defined in terms ofpython-graphblas
internals rather than the public API, and I presume the internals could be subject to change at any time. (In fact, their location has changed since I installedpython-graphblas
.)Is there a better way of doing this that I’m missing, which doesn’t rely on patching the internals? Is this use case something that
python-graphblas
supports in principle? Any suggestions would be welcome. Thanks!Asides:
SelectOp
s aren’t compatible with my UDTs becauseSelectOp
doesn’t implement a_compile_udt
method. Not sure if this is intentional.IndexUnaryOp
working without overridingfind_opclass
, since that function isn’t subclass-friendly. There were a other few places (e.g.,_build
methods) where I needed to define patched classes because of strict type checks that didn’t accommodate subclasses.Beta Was this translation helpful? Give feedback.
All reactions