New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Decide how to package Brian2GeNN #53
Comments
I love the idea of such a simple installation, although I think it's something you'd have to be pretty careful about - making sure that it really is robust. Also, might it cause a problem for people who have an existing GeNN installation? |
I agree that we should be careful about it. I think combining it with an exisiting GeNN installation should be fine, we already have a Brian2GeNN preference that can be used to specify the location of GeNN and which overwrites a |
I can see the appeal of a simple install. On the flip side, while future proofing against future changes in GeNN it would also exclude having the benefits from such changes. In the balance, I think, however, it's probably best to package it in because keeping everything compatible at all times would be tricky. |
Yeah, that's right. I modified GeNN a little while ago to compile all the non-common user model and code generator stuff into the current directory. Like Thomas says, you just need to compile the common libgenn stuff and put it somewhere system-accessible, like /usr/local/lib64 or something - although it's currently just built in the GeNN directory. |
The aim was to put more of the code generation code inside this standalone library, so one didn't have to keep recompiling the code generation code with each project, but I ran into a few problems - with what I can't remember. Maybe I can have another crack at it one day. |
Don't worry about that. Being practical, the current solution should work well with the packaging Marcel proposed. So no need to revisit this now. |
So, I sneakily did a Brian2GeNN release, because you cannot really test the packaging/upload without doing a release... As you can see from the commit history, there were quite a few problems. However, it worked out in the end. we do now have a source code release on pypi (i.e., you can install with pip but then need a correctly configured installation of GeNN + The only thing that did not quite work out: I did not manage to create the conda package with pre-compiled libraries for OS-X on travis. The reason is quite simple, for Windows and Linux I can download and install a minimal version of CUDA for the build process, but the minimal installation with the CUDA installer for OS-X still installs the full SDK with > 1GB -- this takes too long on the build server. So unfortunately we'll need someone with a Mac (ahem, @tnowotny maybe :) ) to manually build the conda packages for OS X. It shouldn't be very complicated: you need to have anaconda (or miniconda), and install the cond-build package ( Could you maybe try (and have friends&family try) the conda packages and/or the pypi source package on your machine and see whether it works for you? To recap, to install from source (needs installed GeNN and
To install via conda (the
If GeNN is installed with the package (i.e. if you use the conda install), then this version of GeNN will be used, otherwise you need to define Well, if all this works fine, I think we are finally ready to do an official 1.0 release 🎉 ! |
FYI: This is no longer necessary, I did another release (1.0rc4) which includes the fix. Oh, unrelated to the packaging (I think), but maybe interesting for @tnowotny , @jamesturner246 , @neworderofjamie & co.: it seems that Brian2GeNN does not work on Windows for the latest version of GeNN from the development branch due to linking issues. I did not look into this in detail, but I think it is rather a GeNN-internal issue. For error messages, see here for example: https://ci.appveyor.com/project/brianteam/brian2genn/build/1.0.156/job/scyjx6bblbqmaxty (note that this runs GeNN in CPU-only mode, as this is all that we can test on these servers). |
Good spot - I was planning on bringing my Windows laptop in tomorrow for other reasons but I'll figure out what's going wrong! Do you know what version of Visual Studio the AppVeyor instances run? |
It's configurable but in our case we use the default image which comes with Visual Studio 2015. |
If you could try re-running the tests that would be great! Thinking about it, I suspect it was something obvious |
We have test runs in the queue, they will pick up the latest state of the development branch automatically. Results should be in in 30 minutes or so. |
I tried building the conda packages on my mac. It fails with
I looked in to the |
BTW: |
Ah sorry, my bad, I forgot about this. This is using fancy "git submodules" which allow you to have a reference to another git respository within a git repository (so we can always update the GeNN version). You'll have to run |
Just for future reference: if you clone |
hum ... getting compilation errors now that look like genn bugs:
But also:
which seems like a straightforward name mis-spelling ...? |
Well our automated testing is testing that same code on your Mac so I don't think those are GeNN bugs per se. Perhaps some Clang compiler flags are being set wrong? Those look like they could be C++11 issues. |
this is the GeNN 2.2.3 release I think ... was that tested on my Mac? |
Command issued by make was |
I remember some problem like this, I think it might be mixing up gcc and clang. Could you check if it works if in the |
hum ... doesn't seem to make a difference |
But I agree with @neworderofjamie that it looks like issues with the C++11 standard ... even though |
hmm, let's narrow it down:
does it work? This is the same what gets called during the package build process. If this works, then the build environment is to blame. |
yes ... this works ... |
Ok. I still have a hunch that the |
ok ... I think we have overcome this problem ... the next one is this:
I don't understand too much of it but it seems it is looking for more recipe files |
Hmm, not sure about this. It might be that the little Python script I wrote is not very robust with respect to paths... When we build packages, we run it from the main directory (i.e. with |
There are some anomalities with my anaconda install
where |
I have no idea what is going on there... Maybe last try: update conda/conda-build with Sorry for this annoying remote-debugging, if only Apple would allow me to have an OS-X virtual machine... 🙄 |
hi again ... I tried all that and also installed anaconda for python 3.6 (I had the 2.7 version before) but no change. I am worried that this nonsense might be causing the problem:
But I have no clue where this mangled channel url is coming from. Should have been
|
This indeed looks weird and I also have no idea where it could come from... I know all this is annoying, but maybe you could do a fresh anaconda/miniconda installation in parallel? If you manually change the PATH within your session to use that one instead of the old one, it would not affect your main installation. At least we'd be sure that your installation is to blame if it works with the new one... |
can do ... in the meantime ... can you have a look whether the packages I made and sent you by email are any good? All I did was removing the test section from the recipe. That shouldn't affect the results I would have thought. |
ok ... tried miniconda ... same problem with testing as before ... also the weird channel name. |
I did not receive any email with packages, I'm afraid... I'm not too surprised that it works if you remove the testing, it seems that on your system for some weird reason it does not find the package it just built during the testing phase, leading to all kind of error messages about missing files, etc. I'm happy to upload any packages you send me -- I cannot really test them, though, as you need a Mac for that. |
Oh, forgot about it: @neworderofjamie the tests on Windows against GeNN's development branch are now working fine, seems your commit was all that was needed. |
I sent the email to Marcel Stimberg marcel.stimberg@inserm.fr |
It is. Maybe it landed in INSERM's spam filter, I might get a notification about it later... |
Oh boy ... I am not sure we should be that patient ... I have added and pushed the packages to the brian2genn repo |
Thanks for the packages, I uploaded them to the anaconda server.
And then try running one of the example scripts with it? (Maybe put an |
Hum ... it almost works ... there is a problem with |
PS: If I define the appropriate |
If brian2genn comes with its own GeNN version, it should use it (you can add the line The |
Yes ... this seems to be the case. With the debug on I get
So it is planning to use the right genn install but somehow it doesn't get passed into the genn-buildmodel.sh script |
for all it's worth I have implemented GENN_PATH auto-detection for Mac in the genn-buildmodel.sh script in our GeNN development branch ... but maybe too late to add it to GeNN 2.2.3 |
I fixed the
again? If it still fails during the testing phase, you can now do:
instead of editing the |
I compiled the packages and tried to send to the google address. But it is returning the email because the "attachments pose a potential threat". I will send you a link instead. |
I uploaded the packages again, could you re-try installing and using them? |
It seems to work now! I have run the CUBA.py example. The only error I get is
I suspect this must be unrelated? The brian2genn stuff has most certainly finished at that point ... |
Hey, great! I think you simply get the error because you are running it in an environment without matplotlib (it's an optional dependency for Brian, all we do is to import it into our namespace...). |
I suppose we need not worry about that too much? Or do you think naive users would run into the same problem if downloading brian2genn was their very first exposure to brian 2? |
I think it is fine, we should maybe document it a bit more (we do mention it in the Brian docs but not in the Brian2GeNN docs). I'll add a note in the docs, and, well I think then we are ready to do a proper 1.0 release, no? @thesamovar Could you maybe also have a go at trying to install it under Windows? >>> import brian2genn
>>> brian2genn.example_run() We do this during the package build process as well (but with |
Yes, I think we should be ready. Exciting. We can then think about publication again. I think @thesamovar had a point that a classical paper still carries some weight. The new thing in it would have to be comparative benchmarking (aside just describing what it is all about and how it roughly works). |
I'm resurrecting this old thread, because I was looking into releasing brian2genn 1.2 (following GeNN's 3.2 release). As a reminder, we currently ship conda packages in our This is kind of cool, but it comes with a few inconveniences. The first is that the packaging process is only half-automated (e.g. OS X packages have to be built by someone) and we cannot automate it further by putting things on conda-forge, since we need the CUDA compiler. The second inconvenience is that these packages simply do not always work due to incompatibilities between compilers/libraries etc., see e.g. #59. I think all this is probably fixable, but I wonder whether it is really worth it for saving users one download + the setting of an environment variable... What do you think? If we remove this, users would have to install GeNN as detailed in the GeNN installation instructions (including setting |
To be honest, I think it is probably more future proof and less support-request heavy if we require users to install their own GeNN? |
@tnowotny I think I agree... I'll open a PR that makes the change (basically: deletes a lot of stuff). |
I think we are ready to publish a first official (beta?) release of Brian2GeNN, it now reasonably well detects unsupported features and it runs non-trivial examples like Kremer et al. (2011).
I wonder what is the best way to package it, though. Brian2GeNN does of course depend on GeNN, but it would not make much sense to specify this dependence on the Python level, since GeNN is not a Python package (we won't upload it to pypi). In principle, we could create a conda package for GeNN, but given that it would not do much except for copying the files, I am not sure it is worth it.
One option I was considering: include the GeNN source code itself into Brian2GeNN via a submodule, and ship it alongside Brian2GeNN in pip/conda packages. I think this would be the most user-friendly option (a simple
pip install brian2genn
orconda install -c brian-team brian2genn
would work), and it would also avoid potential conflicts with future versions of GeNN.There is one problem with this solution, though: both pip and conda allow the installation of packages system-wide (even though it is uncommon for conda), so GeNN could end up in a location not writeable by users without administrator rights. During the compilation of GeNN projects, some files will be compiled to the source directory, so this would fail. Could we pre-compile these files when we create the conda package (let's not worry too much about pip, this could remain "semi-automatic", i.e. you'd still have to install GeNN independently), or do any of these files have to be re-compiled in a project-dependent way?
The text was updated successfully, but these errors were encountered: