GPU enhanced Neuronal Networks
Tips for converting Allen V1 Glif network from NEST to PyGENN. #543
Replies: 11 comments 28 replies
-
|
We actually had a student work on implementing this model over the summer at https://github.com/williamsnider/v1_project which could be a good starting point. He got as far as implementing all the neuron models (https://github.com/williamsnider/v1_project/blob/main/compare_GLIF.ipynb) and got pretty close to parsing the model description and simulating the model (https://github.com/williamsnider/v1_project/blob/main/complete_model.py). I think the best approach (and the one he took) is to directly transfer the structure of populations to GeNN as number of populations isn't really a limitation. One thing to bear in mind is that GeNN doesn't currently support back-propagation through time (which is what Franz Scherr was using) so would have to figure out some other way to train the model |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for all of these helpful links @neworderofjamie. Luckily our lab is uninterested in back-propagation through time, which might make things easier for me. We were aiming to use a local learning rule like STDP, and if that is too hard to do on large-scale simulation using GPU use simpler potentiation and depression rules. Probably the network won't learn as well due to unsupervised learning rules, but we are still interested in the simulation results from a network even if it learns less effectively to do an image change task. I don't expect that a fully developed adult V1 model would be very plastic, but I would like to do simulated developmental pruning using STDP and lateral inhibition to make WTA all circuits. When I consult the Allen V1 connectome using Sonata/HDF5, it looks like all of the synaptic weights belonging to the measured network are static, so I will need to contrive in a meaningful location for the plastic synapses. The overall approach I will take will be bio-inspired but not biologically faithful, so I will probably contrive learning rules in neurons that are not supposed to be plastic. I will investigate this MA student's work. Thanks for the leads. |
Beta Was this translation helpful? Give feedback.
-
|
I succeeded in getting their code to run with the CUDA backend. I am just waiting for it to complete, its a big job. |
Beta Was this translation helpful? Give feedback.
-
|
I forked the repository, added in some fixes to increase reproducibility, and added in a progress bar The problem is while still trying to build the model, the code hits some very slow pandas data frame processing. It might be faster to eschew data frames beyond this point. Maybe there is a faster way. Progress = 0.2516% - e5Rbp4_0 to i23Htr3a_5 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @williamsnider, I am really interested in reproducing aspects of your GSoC It is taking a long time for me to do the data frame processing to generate files used by I wonder if you have access to a free file hosting like cloudstor or an institution's paid DropBox, and if so would you be able to share pickled files corresponding to the processed data frame, just before GeNN model is instantiated? If so that would be awesome. I will make sure to cite any pre-pubs or pubs associated with your GSoC work. As an example, I have used here: https://github.com/russelljjarvis/getConnectomes/blob/main/load_allen_measured_model.jl#L9 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @williamsnider, I was also wondering how to change the filehttps://github.com/williamsnider/v1_project/blob/main/complete_model.py so that it executes LIF cell models, instead of GLIF cell models.. I may have answered my own question, it looks like I would change. An argument to a method https://github.com/williamsnider/v1_project/blob/main/utilities.py#L11-L134 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @russelljjarvis, I'm glad you're using the repo! I have been working on an improved script build_model.py to build the model V1 model, which is a work in progress. Specifically, right now it just has the V1 nodes and V1 to V1 edges / synapses populations. It currently builds when I use the "SingleThreadedCPU" backend, and I'm trying to optimize it to build with the "CUDA" backend without running out of RAM. For your purposes this script may be useful since it is a lot clearer than the complete_model.py and also reduces the size of the pandas dataframes so the wrangling is faster. To answer your latter question, "GLIF_models.py" contains the 5 different Allen GLIF models, where GLIF1 is LIF. If you import GLIF1 you can use it when you do model.add_neuron_population(...). Hope this helps |
Beta Was this translation helpful? Give feedback.
-
|
Hi @williamsnider, this is awesome. Thanks for all of this. I wonder if I should PR back any changes? It would be cool to have something that checks if the Allen/Sonata files already exists, and if they don't they get downloaded. Another minor thing that could help a little is I really appreciate all of the work you are doing! Thanks :) |
Beta Was this translation helpful? Give feedback.
-
|
In the printed progress, I put a newline character For example "Progress = {}%", only printed once but everything to the right of it was visible because of refreshing, I did it like this print(
"Progress = {}% - {} to {}".format(
np.round(100 * count / len(v1_model_names) ** 2, 4), pop1, pop2
),
end="\n",
)Also, the parallel map is compatible with progress bars via a hack I am not running this code, on HPC, so I set the max number of workers to a lower bound, at unlimited number of workers the code fails on my machine: try:
with multiprocessing.Pool(8, maxtasksperchild=8) as pool:
list(tqdm(pool.imap_unordered(make_src_tgt_df, items), total=len(items)))
except:
list(tqdm(map(make_src_tgt_df, items), total=len(items)))Finally, I used the pip install tool black to format the code to make it more congruent with PEP8 with minimal effort, I am only noting this in case it looks like I wasted a lot of time making arbitrary format changes, and because the code has not changed in substance, its only cosmetically different. |
Beta Was this translation helpful? Give feedback.
-
|
I agree build_model.py is way faster, thanks! |
Beta Was this translation helpful? Give feedback.
-
|
I now have this directory full of GeNN code:
Assuming I also provided the headers to the cpp call? ~/git/v1_project2/GeNNModel_CODE$ The model seems to be able to partially build but I think it ultimately hits a memory failure on my CUDA laptop (32GB of system RAM, GPU memory unknown).
|
Beta Was this translation helpful? Give feedback.
-
|
The connection matrices aren't in the GeNN code, they're loaded at runtime from the Python data. |
Beta Was this translation helpful? Give feedback.
-
|
Changing the jobs argument to 1, I am still getting the same error. gcc: fatal error: Killed signal terminated program cc1plus I checked dmesg and it looks like I ran out of some kind of memory. Do you think that decreasing the model size even more will free up memory? [29360.557086] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=cc1plus,pid=21827,uid=1000 |
Beta Was this translation helpful? Give feedback.
-
|
This is a problem with the compiler running out of memory - you could try on a machine with 64gb of memory but really it needs further investigation on our side. |
Beta Was this translation helpful? Give feedback.
-
|
Cool. I have simply switched to SingleThreadedCPU backend on the smaller subset of V1 only model. If the number of simulated time steps is an integer, this model evaluates, which is good! I will try to plot the resulting raster plot. For short run times the model outputs no spikes, as evidenced by: This is my different version of build_model.py print([v.spike_event_times for v in pop_dict.values()])outputs a list of Nones. I think this is because the file build_model.py doesn't define netstimulation artificial inputs it is just a prototype for building the model, for executing the model, I would need to combine code from the other slower run_model.py file like this: pop_dict[lgn_model_name].set_extra_global_param("spikeTimes", spike_times)
print("Added {}".format(lgn_model_name))Currently, I have glued together the LGN code that receives spikes stored from disk, with the current approach in build_model.py that builds a small network more quickly. I think I am getting an error about badly trying to connect together populations of different sizes (LGN versus subsampled V1). This is the error message I am getting: Has anyone had luck running a scaled-down version of the NEST model? |
Beta Was this translation helpful? Give feedback.
-
|
Looks like one of the neuron variables you're initialising with an array/list isn't the correct size. I'm guessing this could be an issue with the Couple of other things:
|
Beta Was this translation helpful? Give feedback.
-
|
Cool, thanks for the pointers!
|
Beta Was this translation helpful? Give feedback.
-
|
In #554 I've just implemented a workaround for the slow build times/huge memory usage. Have a go on that branch, passing |
Beta Was this translation helpful? Give feedback.
-
|
Okay I merged in the compiler_friendly_error_handling branch onto my local GeNN repository. I am rerunning the model with that GeNN installation patched via |
Beta Was this translation helpful? Give feedback.
-
|
I am still getting this: Traceback (most recent call last):
File "/home/rjjarvis/git/v1_project2/snider.py", line 674, in <module>
model.load(
File "/home/rjjarvis/git/GeNN/pygenn/genn_model.py", line 664, in load
pop_data.load(num_recording_timesteps)
File "/home/rjjarvis/git/GeNN/pygenn/genn_groups.py", line 591, in load
self._load_vars(self.neuron.get_vars())
File "/home/rjjarvis/git/GeNN/pygenn/genn_groups.py", line 263, in _load_vars
var_data.view[:] = var_data.values
ValueError: could not broadcast input array from shape (17383,) into shape (2250,) |
Beta Was this translation helpful? Give feedback.
-
|
did you try placing a print/using a debugger where I suggested to figure out which population was causing the issue? The fix was only for compile times with the CUDA backend |
Beta Was this translation helpful? Give feedback.
-
|
Sorry yes, I did do that. The last printout from that line of code is |
Beta Was this translation helpful? Give feedback.
-
|
@russelljjarvis I have done a bit more work on this. https://github.com/neworderofjamie/allen_v1 can now simulate the full V1 model (I haven't fully checked the output but it looks good) using this branch of GeNN. Performance is pretty good (0.6s to simulate 1s of realtime on my workstation) although load time is still a few minutes. |
Beta Was this translation helpful? Give feedback.
-
|
But its clear you must mean this branch |
Beta Was this translation helpful? Give feedback.
-
|
Ahh, sorry about that. I changed my mind on one of modifications I made to GeNN and deleted the old branches. That is indeed the correct branch to use. If that gets deleted it will be because it's been merged into master! |
Beta Was this translation helpful? Give feedback.
-
|
Great, no worries. I like it how you have made an awesome (sort of inadvertent) replacement Sonata/HDF5 reader that is fast, and flexible. I think one reason of many reasons your code is fast is because it retains the low-precision UInt type of the cell indexs, and you defer casting the HDF5 generator to a list until you populate the data frame with it (this is just speculation, I am not 100% sure why the pre-processing job is so much faster now). I am not sure if it is worth trying the model on just any Nvidia GPU, should jump to a HPC Nvidia GPU, which will likely have more on board RAM. I will flag someone else in my lab who should be aware of this new approach to simulating the Billeh V1 model. |
Beta Was this translation helpful? Give feedback.
-
|
Hah, it might have been better to learn how to use pySonata but doing it myself wasn't too hard. Keeping everything in pandas i.e. C++ is definitely the reason why it's much faster - anything going through Python lists will be too slow and too memory-hungry. You should be fine on any GPU really - this does need around 32GB of RAM to load but it only actually 2GB of GPU memory. |
Beta Was this translation helpful? Give feedback.
-
I think the branch must be merged into master now, as I can no longer find the branch. |
Beta Was this translation helpful? Give feedback.
-
|
I got pretty close. It built the model. There is a populated directory called: v1_point_CODE ~/git/jamie$ python run.py Then it errors and quits It looks like an error with gcc version on my end. |
Beta Was this translation helpful? Give feedback.
-
|
Yep, CUDA has fairly strict ranges of supported compilers - see https://docs.nvidia.com/cuda/cuda-installation-guide-linux/ for the latest version |
Beta Was this translation helpful? Give feedback.
-
|
Also if I build the docker file I get this: |
Beta Was this translation helpful? Give feedback.
-
|
I can't compile the cuda samples. So I have to figure out how to downgrade gcc make |
Beta Was this translation helpful? Give feedback.
-
|
I found out I needed to install an a cuda driver extension for containers. Now returns +-----------------------------------------------------------------------------+ |
Beta Was this translation helpful? Give feedback.
-
|
Yep, as the readme says:
|
Beta Was this translation helpful? Give feedback.
-
I wonder if anyone has experience running the Allen V1 "measured" connectome using GENN or has any ideas about the best approach to doing this?
The files that specificy cell type and wiring are shared on DropBox here
These connectomes are encoded in Sonata a HDF5 file format. I have divided this network into smaller populations, and I will probably need to start using SynapseMatrixConnectivity::SPARSE as a container of connection matrices in GENN.
I wonder if I can build a layered network, by chaining together multiple connection matrices? My idea is that LGN (excitatory and inhibitory), and V1 (excitatory and inhibitory), would make up four different populations, and I would use multiple connection matrices (as layers) to connect together the various cell populations.
I am aware of this work, which I believe was implemented using PyTorch tensor types: https://www.biorxiv.org/content/10.1101/2021.11.17.469025v1.full
I would like to do a similar simulated experiment but using GENN as the simulator instead.
Beta Was this translation helpful? Give feedback.
All reactions