Single dict file for all c files

[Christian-B]

Instead of having lots of separate dict file to hold log messages have a single one.

As suggested by @rowleya and @alan-stokes

On a full make the file should be restarted.

Advantage:
Easier to use.

Disadvantage on partial rebuilds you would have to only add new values even if all that changed is the line number. This will cause the file and index numbers will grow over time.
(But on the first full make this is removed)

[rowleya]

I wasn't keen because of the disadvantage. There is also the issue of user builds and/or multiple repositories; imagine a situation where you have released library A with binaries and library B with binaries and now want to use binaries from both.

[Christian-B]

If you release different librays at different times you are playing with fire!

[rowleya]

A few other ideas include:

• Load the dicts in to SDRAM during binary loading. That way the dict is immediately accessible but of course takes space (worth checking how big as 1MB total might be worth it).
• Keep a map of dict filename in SDRAM. This uses less space but means it only makes sense to the machine that loaded it.
• Extend the name of the binary in SARK / SCAMP. This means my script wouldn't need the user to type the filename.

[rowleya]

Note, each of the sPyNNaker dict files is about 44K and you only need one copy per binary which is unlikely to be one per core.

[alan-stokes]

as christian pointed this to me (given my mass of notifications that I've not dealt with during my deep dive).

1. It was suggested by myself, and @rowleya didn't like it when i mentioned it to him privately. I only mentioned it to @Christian-B whilst discussing something linked to this. So saying suggested by @rowleya is majorly unfair to him. lol

2. load dicts into sdram. I dislike this idea quite a bit. As your losing sdram for a ybug support.... and even if it wasnt for ybug support, its just losing sdram. yuck. Your saying 16 * 44kb which is 704kb for stuff it shouldnt need to store......

3. Keep a map of dict filename in SDRAM. smaller issue, but still yuck coz of the original issue. We have a host machine that has to give life support to spinnaker anyhow, so keeping stuff like this in host is obviously the better solution. getting ybug to read them is the issue.

4. Extend the name of the binary in SARK / SCAMP. fine, but you've got the question of how big is too big, and this does not solve the issue of telling ybug where the dicts are.

least having one location which ybug knows of (aka its own repo, aka spinnaker tools, aka where we dump all our libraries already......) would mean you dont need to hand it a set of dict paths, as they are spread over at least 4 repos currently (spynnaker, Fec, nengo, gfe) and as we add more front ends, this is going to expand. yuck.

the issue of binary name to dict is a nasty one, which is why i thought we could just avoid the issue with 1 monster dict. esp as it only needs a unique identifer, which can be whatever size we want/need. The issue of outdated logs etc when local compiling is a wee issue, but given a clean up ability with full recompile, which lets be honest happens a lot and is a good idea everytime you pull.... id call it not that bad a issue.

have we ever really released 1 module on its own and not had it back fire when peeps get mixed up? all our releases are often brand new sets which are not backwards compatible..... and when it does happen. just tell them to do a automatic_make.sh and all is good..... and if its pip based...... well then those peeps are highly unlikely to be grumbling over iobuf. they dont get to recompile binaries at all anyhow. Heck if you want to do that, why dont we have the repo seperate ones, and during setup it merges them into spinnaker tools. so that means ya never got this issue???

[Christian-B]

Loading the dict into sdram is bad as it slows down both DSE and Extract iobuffers

[rowleya]

As an example, say sPyNNaker plus GFE plus user code that uses GFE...

The dict is always next to the binary, so if you build a map as you find the binary to load, you can keep this somewhere on the host e.g. a database.

Note though that we encountered an issue recently where the dict files are a pain; if a user runs something then they want us to help debug it but on our own machine, you probably won't have the right dict files. Christian and I had this exact issue just the other day. At that point SDRAM is a useful option (and it is rare that it will be 16 x 44K).

[alan-stokes]

so that kinda makes my argument for me @rowleya. If you had one centeral place to get these from. it would be moving one file over if you wanted to debug someones elses stuff. instead of a ton of them. I agree sdram is the cleanest in this point. But its just wasting sdram for support which is rarely used, and only used with inside the lab to be honest and so im really not hot for that. esp as that would mean putting debug statements in your code can affect the mapping of your network. that sounds really bad.

[andrewgait]

Have we reached any consensus on this issue?

[dkfellows]

If a user wants us to help debug their running binary that they built themselves, they probably have to send us the .txt file anyway (or we can't understand the code). Requiring the mapping dict too isn't onerous. It's just like supporting user C code on host, where figuring out a random running binary is hard!

The one real problem is we don't get ybug to apply the dict for us. Fixing that would require writing Perl...

[Christian-B]

I have decided to reactivate this with #567

This will allow for a python based ybug with conversion built in which I feel is a bigger advantage than that lost because of mixing two builds. Note partial rebuilds are possible as long as you do not rebuild FrondEndCommon and even that if there is a need we can work around.

[Christian-B]

fixed with SpiNNakerManchester/SpiNNUtils#114

[Christian-B]

done