Routing table tools #211

mundya · 2015-12-16T16:41:25Z

Adds some tools for working with routing tables.

mundya · 2015-12-18T18:08:25Z

rig/routing_table/minimise.py

+        merge = get_best_merge(routing_table, aliases)
+
+        # If there is no merge then stop
+        if merge.goodness < 0:


This should be if merge.goodness <= 0

mundya · 2015-12-19T12:00:59Z

@mossblaser I think this is ready for review/would benefit from your thoughts. Phase 2 of routing table minimisation may come along in a later PR.

mundya · 2015-12-20T12:50:06Z

~~@mossblaser - I'm tempted to rewrite the minimiser to allow it to work with routing trees as well so that I can make use of default entries.~~

Scrap that, for the moment I'm going to argue that the likelihood of using default entries having any serious benefit when also using minimisation is slim. I'll re-address this later if necessary.

mundya · 2015-12-20T23:14:38Z

Some stats for anyone following this:

Table name	# entries	# unique routes	Time to minimise (target length=0) / s	# minimised entries
pynn1	2005	220	14.74	566
pynn2	2007	210	15.98	527
nengo_1739	1738	28	0.54	38

These are just some routing tables that I had lying around - I'll go hunting for a larger dataset when this floats back to the top of my priority queue.

That said, this appears to minimise routing tables somewhat effectively.

mundya · 2015-12-20T23:32:37Z

With target_length=1023 rather than 0.

Table name	# entries	# unique routes	Time to minimise (target length=1023) / s	# minimised entries
pynn1	2005	220	2.71	1017
pynn2	2007	210	3.40	1013
nengo_1739	1738	28	0.29	955

mossblaser · 2015-12-21T09:48:46Z

Scrap that, for the moment I'm going to argue that the likelihood of using default entries having any serious benefit when also using minimisation is slim.

I'd certainly be interested in seeing some "ball-park" numbers as to what you lose from not using default routes. I vaguely remember the southampton guys mentioning this sort of thing too...

These are just some routing tables that I had lying around - I'll go hunting for a larger dataset when this floats back to the top of my priority queue.

I think this would be worth a look for more up-to-date nengo things since you've changed a lot of late...

That said, this appears to minimise routing tables somewhat effectively.

Yes indeed! Particularly like the performance numbers when you just say "stop at 1024". On this note we should really work out what this number is by asking the machine... It does seem to me, though, that there is little incentive to do so since "sharing" a routing table is such a liability...

On this note, would it be worth including this tool as part of the P&R wrapper?

mossblaser · 2015-12-21T09:53:27Z

rig/place_and_route/utils.py

@@ -277,6 +277,10 @@ def build_routing_tables(routes, net_keys, omit_default_routes=True):
        If a routing tree has a terminating vertex whose route is set to None,
        that vertex is ignored.

+    .. note::
+        :py:func:`~.build_and_minimise_routing_tables` can be used to get


~~"can be used to minimise routing tables produced by this function."~~

~~I'd include an example too (especially if you need to turn off default-route omission!)~~

I am, of course, an idiot who should read more carefully...

mundya · 2015-12-21T09:58:34Z

I'd certainly be interested in seeing some "ball-park" numbers as to what you lose from not using default routes. I vaguely remember the southampton guys mentioning this sort of thing too...

I have no numbers, but as this algorithm generates entries with large numbers of Xs my gut feeling is that even if a routing entry which could be default routed wasn't merged with anything else you probably wouldn't be able to remove it from the table anyway.

I think this would be worth a look for more up-to-date nengo things since you've changed a lot of late...

Again, no numbers, but this is sufficiently effective that the larger circular convolution models actually fit on the machine.

On this note we should really work out what this number is by asking the machine... It does seem to me, though, that there is little incentive to do so since "sharing" a routing table is such a liability...

I'd agree that sharing routing tables is a Bad Idea (:tm:) unless you can guarantee that the entries which you have to share with are higher priority than the entries you want to minimise and that the unminimised table (including the other entries) is orthogonal.

+1 for asking the machine how any entries are available! Hard-coding 1023 makes me feel very unhappy.

On this note, would it be worth including this tool as part of the P&R wrapper?

I'm happy to do that provided we have a flag to turn it off and document thoroughly... but of course these are things you'd do anyway ;)

mossblaser · 2015-12-21T10:03:12Z

Again, no numbers, but this is sufficiently effective that the larger circular convolution models actually fit on the machine.

Already got a nengo prototype running? Super; I'd call that better than numbers ;).

+1 for asking the machine how any entries are available! Hard-coding 1023 makes me feel very unhappy.

How should this information propagate? Via the P&R infrastructure (e.g. constraints) or via a side channel (e.g. directly passing in a system-info or a raw dictionary?)

mossblaser · 2015-12-21T10:04:55Z

rig/place_and_route/utils.py

+        could not be produced.
+    """
+    # Build the tables and then minimise them
+    tables = {


Due to the CPU-bound nature of the problem and trivial parallelism available here it may be worth just dropping in: https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers as a quick and easy speedup.

mundya · 2015-12-21T10:12:42Z

How should this information propagate? Via the P&R infrastructure (e.g. constraints) or via a side channel (e.g. directly passing in a system-info or a raw dictionary?)

I think it feels somewhat similar to the SDRAM constraint, so I guess via the P&R infrastructure... But I have no strong preference. I was secretly hoping you'd push the API for this into shape ;)

mossblaser · 2015-12-21T10:13:14Z

rig/routing_table/minimise.py

+"Ordered Covering"
+==================
+
+The algorithm implemented here, "Ordered Covering", provides the following


If we're likely to have multiple algorithms available for minimisation (just as we have them available for P&R) it may be a good idea to rename this file to reflect the name of the algorithm and then provide an alias (as you have done) in the top level. Will the different algorithms be sufficiently overlapping in compatibility that this would be a reasonable idea?

Yes, this is probably a reasonable idea.

And is done.

mossblaser · 2015-12-21T10:18:37Z

How should this information propagate? Via the P&R infrastructure (e.g. constraints) or via a side channel (e.g. directly passing in a system-info or a raw dictionary?)

I think it feels somewhat similar to the SDRAM constraint, so I guess via the P&R infrastructure... But I have no strong preference. I was secretly hoping you'd push the API for this into shape ;)

Yeah, the other difficulty is that no vertex is realistically going to consume routing table entries so putting this as a resource seems slightly awkward. Doing it via constraints also seems a good idea but only if this module were a part of the P&R module. Since I'm still unsure whether this belongs in P&R or separately... Doing it via a dictionary seems like by-far the cleanest solution when you look at the function in isolation though...

I'll continue to think about this...

mossblaser · 2015-12-21T10:20:18Z

rig/routing_table/minimise.py

+==================
+
+The algorithm implemented here, "Ordered Covering", provides the following
+rules:


If I understand correctly, these rules are written from the perspective of something generating routing table entries sequentially(?) rather than as a property of a complete routing table. This may be worth stating explicitly.

I'm not sure I'd use the term "generating", but if the table you receive is in order of increasing generality and you apply these rules when you remove and add entries to the table then the final table, and every intermediate table, is guaranteed to be functionally equivalent to the starting table.

(Note if the starting table is ordered but is non-orthogonal then following these rules will still guarantee a functionally equivalent table but it may not be a small as it could be if you removed the non-orthogonal entries.)

mossblaser · 2016-01-04T11:21:46Z

rig/routing_table/utils.py

+    # for where there are common Xs in the second table.
+    common_xs = get_common_xs(entries_b)
+    try:
+        return all(get_first_match(entry.key) == entry.route for entry in


As much as I like this style; this algorithm would certainly be clearer as a for-loop over the keys with the above function simply being inlined as a nested loop.

mundya · 2016-01-04T15:25:05Z

Not a big change in SC&MP: https://bitbucket.org/spinnaker-low-level-software/spinnaker_tools/branch/include-rtr_free-in-cmd_info

I'll compile and try to get Rig to play along shortly.

mundya · 2016-01-05T11:17:35Z

Would it just be neater to allow minimise to accept a SystemInfo or a dict or an int or None for target_length?

mossblaser · 2016-01-05T11:55:10Z

Would it just be neater to allow minimise to accept a SystemInfo or a dict or an int or None for target_length?

Yes and no. Certainly would be neat but the SystemInfo construct is definitely not something I want leaking everywhere... How strong are your feelings on this matter?

mundya · 2016-01-05T12:51:57Z

How strong are your feelings on this matter?

Not very!

@mossblaser

- Moves routing table constructs into a package - Adds tools for working with routing tables - Adds implementation of ordered covering routing table minimisation algorithm. Includes work and suggestions by @mossblaser

Adds largest_free_mc_block to the ChipInfo (and thus SystemInfo) tuple. To enable backward compatibility ChipInfo now has default argument values. Also adds build_target_lengths function which builds a dictionary of target routing table lengths from a SystemInfo object. This commit includes a new build of SC&MP built from commit b8cdcc2. This commit also fixes a preliminary test implementation and also changes the location of the routing table entry count within the arg1 bits.

Also adds an optional argument to build_and_minimise_routing_tables which specifies which algorithm to use.

@mossblaser

- Add `opposite` property to Routes - Add traversal to RoutingTree - Add routing table minimisation to place and route wrapper Includes work by @mossblaser

Improve minimisation API

Routing table tools

mundya · 2016-01-13T08:20:21Z

Woohoo!

mundya added the work in progress label Dec 16, 2015

mundya force-pushed the routing-table-tools branch 5 times, most recently from 83376dc to f120051 Compare December 18, 2015 17:12

mundya reviewed Dec 18, 2015
View reviewed changes

mundya added the requires review label Dec 19, 2015

mundya assigned mossblaser Dec 19, 2015

mundya added enhancement and removed work in progress labels Dec 20, 2015

mossblaser reviewed Dec 21, 2015
View reviewed changes

mossblaser reviewed Jan 4, 2016
View reviewed changes

Add routing table tools

52f9ade

- Moves routing table constructs into a package - Adds tools for working with routing tables - Adds implementation of ordered covering routing table minimisation algorithm. Includes work and suggestions by @mossblaser

mundya force-pushed the routing-table-tools branch 2 times, most recently from 618ce40 to 4f0435e Compare January 7, 2016 10:09

mossblaser added 2 commits January 7, 2016 10:15

Update Sphinx documentation for table minimisation.

920ae2d

Also adds an optional argument to build_and_minimise_routing_tables which specifies which algorithm to use.

mundya force-pushed the routing-table-tools branch from 4f0435e to 920ae2d Compare January 7, 2016 10:15

mossblaser mentioned this pull request Jan 12, 2016

Improve minimisation API #213

Merged

mundya and others added 2 commits January 12, 2016 16:49

Improve routing table minimisation API

231f691

- Add `opposite` property to Routes - Add traversal to RoutingTree - Add routing table minimisation to place and route wrapper Includes work by @mossblaser

Merge pull request #213 from project-rig/minimize-in-out-pairs

b3c8d67

Improve minimisation API

mossblaser added a commit that referenced this pull request Jan 13, 2016

Merge pull request #211 from project-rig/routing-table-tools

80f35dc

Routing table tools

mossblaser merged commit 80f35dc into master Jan 13, 2016

mossblaser deleted the routing-table-tools branch January 13, 2016 09:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Routing table tools #211

Routing table tools #211

mundya commented Dec 16, 2015

mundya Dec 18, 2015

mundya commented Dec 19, 2015

mundya commented Dec 20, 2015

mundya commented Dec 20, 2015

mundya commented Dec 20, 2015

mossblaser commented Dec 21, 2015

mossblaser Dec 21, 2015

mundya commented Dec 21, 2015

mossblaser commented Dec 21, 2015

mossblaser Dec 21, 2015

mundya commented Dec 21, 2015

mossblaser Dec 21, 2015

mundya Dec 21, 2015

mundya Dec 21, 2015

mossblaser commented Dec 21, 2015

mossblaser Dec 21, 2015

mundya Dec 21, 2015

mossblaser Jan 4, 2016

mundya commented Jan 4, 2016

mundya commented Jan 5, 2016

mossblaser commented Jan 5, 2016

mundya commented Jan 5, 2016

mundya commented Jan 13, 2016

Routing table tools #211

Routing table tools #211

Conversation

mundya commented Dec 16, 2015

Choose a reason for hiding this comment

mundya commented Dec 19, 2015

mundya commented Dec 20, 2015

mundya commented Dec 20, 2015

mundya commented Dec 20, 2015

mossblaser commented Dec 21, 2015

Choose a reason for hiding this comment

mundya commented Dec 21, 2015

mossblaser commented Dec 21, 2015

Choose a reason for hiding this comment

mundya commented Dec 21, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mossblaser commented Dec 21, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mundya commented Jan 4, 2016

mundya commented Jan 5, 2016

mossblaser commented Jan 5, 2016

mundya commented Jan 5, 2016

mundya commented Jan 13, 2016