Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LFP computation on HPC systems #375

Open
3 tasks done
ElisabettaGiacalone opened this issue Jan 23, 2019 · 54 comments
Open
3 tasks done

LFP computation on HPC systems #375

ElisabettaGiacalone opened this issue Jan 23, 2019 · 54 comments
Assignees
@ppodhajski
Labels
Topic_ExpBrain_SimUI Issues related to the Simulation UI use case. Type_Feature Label to identify new/changed features for an existing use case
Milestone
M18: September 2019

Comments

@ElisabettaGiacalone
Copy link
Collaborator

ElisabettaGiacalone commented Jan 23, 2019 •
edited
Loading

Use Case Update

Aspect Detail
Summary LFP computation on HPC systems (Cineca ,Juelich, CSCS) in addition to BBP system
Expert @pramodk
Deadline TBD
Dependencies HPC

Remove anything that does not apply

List of additional/changed features

The new feature will allow to run the following code on the above mentioned HPC systems:

module purge
module load nix/viz/emsim
emsim -i /gpfs/bbp.cscs.ch/project/proj42/home/giacalon/w1syn01/BlueConfig --target sliceb2 --report AllCompartmentsMembrane --voxel-size 1,1,1 --start-time 200 --end-time 1500 --output=sliceb2_9
--sample-point 2494.5,5725.4,5976.8

Tasks

  • LFP computation on Juelich
  • LFP computation on CSCS
  • LFP computation on Cineca

Acceptance Criteria

  • LFP computation on Cineca, Juelich, CSCS systems

Extra Requirements

System

Performance
@ElisabettaGiacalone ElisabettaGiacalone added Type_Feature Label to identify new/changed features for an existing use case Topic_ExpBrain_SimUI Issues related to the Simulation UI use case. labels Jan 23, 2019
@pramodk
Copy link
Collaborator

pramodk commented Jan 23, 2019 •
edited
Loading

@ppodhajski : From simulation side we are deploying all stack via Spack. As this package is only with nix, could you have a look at this? If viz would like to have this deployed via Nix, @tristan0x can help you . Or, if you can package inside spack, it can be deployed with simulation stack.

@ppodhajski
Copy link

Yes, I will have a look,. In the team we had some discussion on Spack, so maybe it is a good opportunity to test if Spack works for viz team.

@jdcourcol
Copy link
Collaborator

@ppodhajski Please deploy with nix first. I think Spack investigation can take a long time and I would avoid to have that in the critical path.

@alex4200 alex4200 added this to the M12: March 2019 milestone Jan 28, 2019
@alex4200
Copy link
Contributor

@ElisabettaGiacalone Please see related ticket #154

@jdcourcol
Copy link
Collaborator

@alex4200 can we have @ppodhajski assigned to that one instead ?
did he accept the invite ?

@alex4200
Copy link
Contributor

alex4200 commented Jan 29, 2019 via email

@ppodhajski
Copy link

accepted

@ppodhajski ppodhajski assigned ppodhajski and unassigned pramodk Jan 29, 2019
@alex4200
Copy link
Contributor

alex4200 commented Feb 4, 2019

@ppodhajski Is there any news regarding this issue?

@ppodhajski
Copy link

with help of Tristan I installed nix on jurecavis (via spack), as it had to be reinstalled after introducing new usage model at JSC. Currently trying to fix the issue when building with nix.

The problem encountered:
patching /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/nix/store/aa8kd6z12mj69yx89m8km66ccs7cdli6-bootstrap-tools/bin/[
stat: Value too large for defined data type

@ppodhajski
Copy link

ppodhajski commented Feb 19, 2019 •
edited by alex4200
Loading

@ElisabettaGiacalone please try executing:
/p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun

@ElisabettaGiacalone
Copy link
Collaborator Author

@ppodhajski I get "permission denied" to access to emsim folder

@ppodhajski
Copy link

@ElisabettaGiacalone please retry

@ElisabettaGiacalone
Copy link
Collaborator Author

@ppodhajski
Now I can access to emsim folder, but when I execute
/p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun
I get:
/p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun: line 7: /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/bin/emsim: Permission denied

@ppodhajski
Copy link

ppodhajski commented Feb 19, 2019 via email

@clupascu
Copy link
Collaborator

I get now
/p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/bin/emsim: error while loading shared libraries: libboost_program_options.so.1.54.0: cannot open shared object file: No such file or directory

@ppodhajski
Copy link

ppodhajski commented Feb 19, 2019 via email

@clupascu
Copy link
Collaborator

/p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun
LD_LIBRARY_PATH to be reported: /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/lib:/usr/local/software/jureca/Stages/2018b/software/binutils/2.31.1-GCCcore-7.3.0/lib:/usr/local/software/jureca/Stages/2018b/software/GCCcore/7.3.0/lib/gcc/x86_64-pc-linux-gnu/7.3.0:/usr/local/software/jureca/Stages/2018b/software/GCCcore/7.3.0/lib64:/usr/local/software/jureca/Stages/2018b/software/GCCcore/7.3.0/lib
Error: the option '--input' is required but missing

-h [ --help ] Print this help message.

-i [ --input ] arg Path to Blueconfig file.
-o [ --output ] arg Path for the output file.
--target arg The circuit's target.
--report arg The name of the report.
--start-time arg The start time
--end-time arg The end time
--fraction arg Specify the fraction [0.0 1.0] of gids to be used
during the computation. Default is 1.0.
--export-volume Will export a floating point volume for each time
steps.

--voxel-size arg The size in each dimension of a voxel in circuit units.
Default is 4.0,4.0,4.0. Must be written in the form:
--voxel-size rx,ry,rz
--volume-extent arg Specify an additional 3d extent for the volume in
micrometers. Default is 0.0,0.0,0.0. Must be written in
the form: --volume-extent ex,ey,ez
--sample-point arg The x y z positions of a sample point. Must be written
in the form: --sample-point x,y,z

@ppodhajski
Copy link

ppodhajski commented Feb 19, 2019 via email

@ppodhajski
Copy link

ppodhajski commented Feb 19, 2019 via email

@ElisabettaGiacalone
Copy link
Collaborator Author

Yes, I get the same output, but I suppose it is the output with all options of the function.
I am going to try with all the required data and I will give you a feedback.

@ElisabettaGiacalone
Copy link
Collaborator Author

Hi @ppodhajski, I tried with a test simulation output, and it works.

/p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun -i /p/scratch/cvsk25/vsk2505/test-190219/BlueConfig --target slice0 --report AllCompartmentsMembrane --voxel-size 1,1,1 --start-time 0 --end-time 300 --output=slice0 --sample-point 3740.8,2398.6,5724.4

@ElisabettaGiacalone
Copy link
Collaborator Author

@ppodhajski Would it be be possible to do the same on Cineca and CSCS systems?

@ppodhajski
Copy link

@ElisabettaGiacalone I am waiting for access rights to CSCS.
Will update the ticket then.

@alex4200
Copy link
Contributor

alex4200 commented Apr 1, 2019

@ppodhajski No news on this item?

@ppodhajski
Copy link

@ppodhajski, Is there any update on the installation on CSCS ?

@ElisabettaGiacalone what path on CSCS should it be deployed?

@clupascu
Copy link
Collaborator

clupascu commented Apr 8, 2019

I think it can be deployed here /apps/hbp/ich002/hbp-spack-deployments/

@ppodhajski
Copy link

@clupascu @elisamarenzi please see it under /apps/hbp/ich002/hbp-visualisation-deployements/emsim/emsim

@ElisabettaGiacalone
Copy link
Collaborator Author

ElisabettaGiacalone commented Apr 17, 2019 •
edited
Loading

@ppodhajski I have just tested that on CSCS with this code:

/apps/hbp/ich002/hbp-visualisation-deployements/emsim/emsim -i /scratch/snx3000/bp000139/w1syn03_testLFP/BlueConfig --target slice2 --report AllCompartmentsMembrane --voxel-size 1,1,1 --start-time 200 --end-time 600 --output=slice2 --sample-point 3740.8,2398.6,5724.4

But I get this error (despite the input is specified) :

Error: the option '--input' is required but missing

-h [ --help ] Print this help message.

-i [ --input ] arg Path to Blueconfig file.
-o [ --output ] arg Path for the output file.
--target arg The circuit's target.
--report arg The name of the report.
--start-time arg The start time
--end-time arg The end time
--fraction arg Specify the fraction [0.0 1.0] of gids to be used
during the computation. Default is 1.0.
--export-volume Will export a floating point volume for each time
steps.

--voxel-size arg The size in each dimension of a voxel in circuit units.
Default is 4.0,4.0,4.0. Must be written in the form:
--voxel-size rx,ry,rz
--volume-extent arg Specify an additional 3d extent for the volume in
micrometers. Default is 0.0,0.0,0.0. Must be written in
the form: --volume-extent ex,ey,ez
--sample-point arg The x y z positions of a sample point. Must be written
in the form: --sample-point x,y,z

@ppodhajski
Copy link

Hi @ElisabettaGiacalone,
I am back to the office after my holidays, please try it now; there was a small permission issue

@ElisabettaGiacalone
Copy link
Collaborator Author

Hi, I just tried on CSCS and it works now. Thanks.

I am going to test it on Marconi.

@ElisabettaGiacalone
Copy link
Collaborator Author

@ppodhajski I have just tried on Marconi, but I think I have the same permission problem as before on CSCS:

[mmiglior@r000u07l02 test-31aug]$ /marconi_work/HBP_CDP2_it_0/bbp-visualisation/emsim/emsim -i /marconi_work/HBP_CDP2_it_0/hippo-mm/test-31aug/BlueConfig --target slice2 --report AllCompartmentsMembrane --voxel-size 1,1,1 --start-time 0 --end-time 300 --output=slice2 --sample-point 3740.8,2398.6,5724.4
Error: the option '--input' is required but missing

-h [ --help ] Print this help message.

-i [ --input ] arg Path to Blueconfig file.
-o [ --output ] arg Path for the output file.
--target arg The circuit's target.
--report arg The name of the report.
--start-time arg The start time
--end-time arg The end time
--fraction arg Specify the fraction [0.0 1.0] of gids to be used
during the computation. Default is 1.0.
--export-volume Will export a floating point volume for each time
steps.

--voxel-size arg The size in each dimension of a voxel in circuit units.
Default is 4.0,4.0,4.0. Must be written in the form:
--voxel-size rx,ry,rz
--volume-extent arg Specify an additional 3d extent for the volume in
micrometers. Default is 0.0,0.0,0.0. Must be written in
the form: --volume-extent ex,ey,ez
--sample-point arg The x y z positions of a sample point. Must be written
in the form: --sample-point x,y,z

@ppodhajski
Copy link

@ElisabettaGiacalone it is fixed now!

@ElisabettaGiacalone
Copy link
Collaborator Author

Thanks, it works now.
We can close the issue.

@clupascu clupascu mentioned this issue Jun 24, 2019
Closed
7 tasks
@antonelepfl
Copy link
Collaborator

Hi @ppodhajski, I would need this fix deployed on PizDaint and Jureca.

@antonelepfl antonelepfl reopened this Apr 21, 2020
@ppodhajski
Copy link

hi @antonelepfl, I am trying to deploy it via spack on piz daint but ran into some compilation issues. will keep you posted.

@antonelepfl
Copy link
Collaborator

Do you have any update on this?

@ppodhajski
Copy link

ppodhajski commented Apr 28, 2020 •
edited
Loading

please try binaries in the folder below:
/apps/hbp/ich002/hbp-visualisation-deployements/spack/softwares/28-04-2020/install/install/cray-cnl7-haswell/intel-19.0.1.144/emsim-1.0.0-vbdt6g/bin

or built with gcc:
/apps/hbp/ich002/hbp-visualisation-deployements/spack/softwares/28-04-2020/install/install/cray-cnl7-haswell/gcc-8.3.0/emsim-1.0.0-lewa5k/bin

These are built with brion version 3.1.0 (https://github.com/BlueBrain/Brion/releases)

@ElisabettaGiacalone
Copy link
Collaborator Author

Hi @ppodhajski, I am trying emsim on Jureca for the first time with the new model, but I get this error:

[giacalone1@jrl12 type_l_test]$ /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/emsim -i BlueConfig --target slice30 --report AllCompartmentsMembrane --voxel-size 1,1,1 --start-time 0 --end-time 300 --output=slice0 --sample-point 3740.8,2398.6,5724.4 WARNING: Time range is clamped to the minimum bound. INFO: Time range is: [300 300] with DT: 0.25 Normalized current sum: -3.43139e-09 INFO: Compartments count: 8300167 terminate called after throwing an instance of 'std::runtime_error' what(): Could not open morphology file /p/project/cvsk25/vsk2516/hpc/entities/morphologies/20180417//011127HP1_-_Clone_0.h5: Unable to open file /p/project/cvsk25/vsk2516/hpc/entities/morphologies/20180417//011127HP1_-_Clone_0.h5 (File accessibilty) Unable to open file /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/emsim: line 11: 31321 Aborted LC_NUMERIC=C LD_LIBRARY_PATH=${HERE}/lib:$LD_LIBRARY_PATH ${HERE}/bin/emsim "$@"

@ppodhajski
Copy link

ppodhajski commented Apr 28, 2020 via email

@ElisabettaGiacalone
Copy link
Collaborator Author

@ppodhajski please, could you deploy it also on Jureca?

@antonelepfl
Copy link
Collaborator

Your deployment on PizDaint works correctly. I tried both paths and they work. Which one do you recommend to use?

@ppodhajski
Copy link

ppodhajski commented Apr 29, 2020 •
edited
Loading

I cannot download some sources on jureca:

[podhajski1@jrc1387 spack]$ wget https://sourceware.org/
--2020-04-29 13:28:49--  https://sourceware.org/
Resolving sourceware.org (sourceware.org)... 209.132.180.131
Connecting to sourceware.org (sourceware.org)|209.132.180.131|:443...

Do you know if I there is some proxy I could use? In the meantime I created a ticket for this issue.

@ppodhajski
Copy link

Which one do you recommend to use?

Both are fine to use.

@mmigliore
Copy link
Collaborator

@ppodhajski I am not sure to understand the wget problem.
Is it the first time that you try to get those files, or you were able to get them before and now it does not work anymore?
In any case, if you and/or @pramodk think it is a problem with the Jureca system, open a ticket by sending an email to sc@fz-juelich.de

@pramodk
Copy link
Collaborator

pramodk commented Apr 29, 2020

Yes, this should go to :sc@fz-juelich.de.

@ppodhajski
Copy link

In the meantime I deployed emsim with brion fixes using an AppImage.
@ElisabettaGiacalone please validate it:
/p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/emsim

@ElisabettaGiacalone
Copy link
Collaborator Author

@ppodhajski it works correctly, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ppodhajski ppodhajski

Labels
Topic_ExpBrain_SimUI Issues related to the Simulation UI use case. Type_Feature Label to identify new/changed features for an existing use case
Projects
None yet
Milestone
M18: September 2019
Development

No branches or pull requests
8 participants
@pramodk @jdcourcol @alex4200 @ppodhajski @clupascu @antonelepfl @mmigliore @ElisabettaGiacalone