New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add update_environment function to manage.sh #170

Closed
wants to merge 923 commits into
base: master
from

Conversation

Projects
None yet
6 participants
@jmbenlloch
Contributor

jmbenlloch commented Mar 13, 2017

Add new function update_environment X.Y to update all packages within one particular IC environment to their latest versions available in conda.

Basically one needs to run:

source manage.sh update_environment 2.7
source manage.sh update_environment 3.5
source manage.sh update_environment 3.6

This is the low-hanging fruit from #166. Opinions?

@jacg

This comment has been minimized.

Show comment
Hide comment
@jacg

jacg Mar 13, 2017

Collaborator

Updating conda-installed packages is not enough: some things we get through pip!

Collaborator

jacg commented Mar 13, 2017

Updating conda-installed packages is not enough: some things we get through pip!

@jacg

This comment has been minimized.

Show comment
Hide comment
@jacg

jacg Mar 13, 2017

Collaborator

I'd also like an update_all function, so that we don't have to do the same thing N times for however many python versions we happend to be supporting at the time.

Collaborator

jacg commented Mar 13, 2017

I'd also like an update_all function, so that we don't have to do the same thing N times for however many python versions we happend to be supporting at the time.

@jmbenlloch

This comment has been minimized.

Show comment
Hide comment
@jmbenlloch

jmbenlloch Mar 13, 2017

Contributor

You are right, I forgot about those, I'll updated the code and also create update_all.

Contributor

jmbenlloch commented Mar 13, 2017

You are right, I forgot about those, I'll updated the code and also create update_all.

@jmbenlloch

This comment has been minimized.

Show comment
Hide comment
@jmbenlloch

jmbenlloch Mar 13, 2017

Contributor

Ok, I have added update_all so we can run only one command to update all environments. Not sure whether awk is available in MacOS, if not it can be changed easily by something else.

Regarding pip-installed packages, it is not obvious how to update them. There is no command to update all packages in pip (pypa/pip#59). If we want to iterate manually over all packages installed by pip we need to know which ones are those, but pip list and pip freeze shows all packages installed (also those from conda).

Contributor

jmbenlloch commented Mar 13, 2017

Ok, I have added update_all so we can run only one command to update all environments. Not sure whether awk is available in MacOS, if not it can be changed easily by something else.

Regarding pip-installed packages, it is not obvious how to update them. There is no command to update all packages in pip (pypa/pip#59). If we want to iterate manually over all packages installed by pip we need to know which ones are those, but pip list and pip freeze shows all packages installed (also those from conda).

@jacg

This comment has been minimized.

Show comment
Hide comment
@jacg

jacg Mar 13, 2017

Collaborator

This whole manage.sh business makes me nervous because

  • I don't see how we can sensibly test it automatically, so we have to test it by eye.
  • There's lots of repetition in there, and because it's never of highest urgency, we're putting up with it and slowly letting it grow into a monster.

The conda env | grep IC, while a great idea in principle, makes me nervous in practice. I just hope that I (and everyone else) remember never to have any non-IC related envs with IC in their name.

As for pip ... hmm, some thoughts

  • What if we just trash whole of pip, and re-install everything. To what extent is pip shared between different conda envs?
  • Iterate over the things mentioned in the pip section of the environment specification. This will probably fail to update dependencies.
Collaborator

jacg commented Mar 13, 2017

This whole manage.sh business makes me nervous because

  • I don't see how we can sensibly test it automatically, so we have to test it by eye.
  • There's lots of repetition in there, and because it's never of highest urgency, we're putting up with it and slowly letting it grow into a monster.

The conda env | grep IC, while a great idea in principle, makes me nervous in practice. I just hope that I (and everyone else) remember never to have any non-IC related envs with IC in their name.

As for pip ... hmm, some thoughts

  • What if we just trash whole of pip, and re-install everything. To what extent is pip shared between different conda envs?
  • Iterate over the things mentioned in the pip section of the environment specification. This will probably fail to update dependencies.

jjgomezcadenas and others added some commits Jul 17, 2017

Use of DataVectors and PmapVectors in cities.
Traffic of arguments in function events_loop for the various cities
expressed more concisely  with the user of DataVectors and PmapVectors
containers.
Split HitCity into KrCity and HitCity
A KrCity computes and writes KrEvents, e.g, point-like events. This
means that the baricenter integrates all the time slices for a given
SiPM. Instead, a HitCity computes and writes HitEvents, in which the
baricenter (or any other position-finding algo) is computed
slice-by-slice. A HitCity is derived from a KrCity since both of them
use the same file_loop() function (which starts from pmaps).
Add traceback after printout if except
Without the exit, the command-line test was passing with pretty greens
when the city was not found (e.g, a captured ModuleNotFound, print
that pytest captures and digests without telling anything and happy
ending in spite of wrong situation). Adding traceback still gives
printout when running in normal mode but provides diagnostics of the problem
Run 10 events by dafault in Penthesilea
The first few events are rejected by selection in the test sample in
Penthesilea, running 10 events guarantees that the city runs fully.
write sipm_ids_and_charges_in_slice with test
this function allows takes an s2sid_peak and slice number and returns
the sipm ids of sipms in that s2si, with their charges at slice number
Add pmaps_functions_c to base.
This is needed to use the corresponding cython function rather than
previous python version.
Modify functions sipm_ids_and_charges_in_slice.
Include in the list only slices with positive charge.
Fix function compute_xy_position
One has to take into account the possibility that the slice results in
an empty SiPM list. This is handled via SipmEmptyList exception.
Remove Cecilia from cities.
Cecilia is removed from cities. The simulation of the trigger will be
done shortly in Diomira.
Add NN (NoNumber) with value -999999
Although it may be argued that we reinvent the wheel, the use of a
numerical NN is quite convenient when writing/reading pytables to/from
disk, as a proxy for NaN.
Cluster takes std (not rms) as argument.
Cluster takes the std (not the rms) as argument. A number of
properties (Xrm and such) are modified accordingly.
Various tests involving cluster fixed.
The change of definition in Cluster (takes std, not rms) requires to
fix a number of tests.
Redefined conf. files for kr_city and hit_city.
hit_city inherits now from kr_city. Config. files modified accordingly.

Alejandro Botas and others added some commits Sep 1, 2017

Alejandro Botas Jacek Generowicz
Raise exception when constructing empty pmaps
Close #315.

Pmap constructors: S1, S2, S2Si, S1Pmt, S2Pmt will now raise an
exception when given empty dicts to initialize. Previously if we had
an empty s2, we would call and initialize an empty S2Pmt --> S2 -->
S12. This is now avoided.

If the constructor is called to build a peaks out of empty
dictionaries, the constructor now raises InitializeEmptyPmapObject
which must be caught.

The general flow of things has changed such that where previously if
there were no s2 in an event, IC would pass around empty s2, s2si, and
s2pmt, Now we just pass around None.

This PR merely creates this exception and catches it throughout the
codebase where necessary.
Allow irene to find ipmt pmaps
This pr will allow irene to make use of the ipmt pmap finders and
writers in order to save ipmt pmaps. I've run this irene in my
notebook a few times and everything looks good so far.

    In order to tell irene to write ipmt pmaps (or not) I had to put a
    flag compute_ipmt_pmaps in pmap_city.conf. This means that
    previous irene config files will not work without first adding the
    flag. This may be kind of annoying for people running productions
    @jmbenlloch @jerenner @gonzaponte. If there is a better way of
    doing this, please let me know, and I will implement it. The only
    alternative I could think of would be to create another city, but
    that sounds potentially even more annoying for you guys.

    I did some time/memory profiling for on Kr data. From run 4446
    with thr_sipm = 1pes and 'thr_sipm_s2 = 2pes' and found after
    running 1k events:
    speed
    1.92 evt / sec with IPMT
    2.06 evt / sec without
    file size
    2.37 x larger than without IPMT
    So, time does not appear to be an issue, but the file size does
    grow substantially...

    @jjgomezcadenas I don't remember if we decided we wanted to put
    the ipmt pmaps in a different group, called IPMT in the hdf5
    file... did we?
Use event numbers from input files for MC
We were computing a new event number in Diomira because in the
previous MC production, each file restarted the event counter to
zero. This is not the case anymore, each file has a unique event
number. Since this was a patch intended to use that old production, it
could be removed now.
@jacg

This comment has been minimized.

Show comment
Hide comment
@jacg

jacg Sep 7, 2017

Collaborator

This is stale: any plan to move it forward, @jacg, @jmbenlloch?

We'll get around to it eventually.

Collaborator

jacg commented Sep 7, 2017

This is stale: any plan to move it forward, @jacg, @jmbenlloch?

We'll get around to it eventually.

@jacg jacg referenced this pull request Sep 7, 2017

Closed

Failure in two diomira tests. #299

Jacek Generowicz and others added some commits Sep 8, 2017

Jacek Generowicz
Deal with 1 hit in voxelize_hits
Closes #328

Previously, the code was crashing when a hit collection contained only
one item.
@jacg

This comment has been minimized.

Show comment
Hide comment
@jacg

jacg Jul 26, 2018

Collaborator

I suspect that this is going to be superseded by #496.

Collaborator

jacg commented Jul 26, 2018

I suspect that this is going to be superseded by #496.

@jacg

This comment has been minimized.

Show comment
Hide comment
@jacg

jacg Jul 28, 2018

Collaborator

I think that this is being superseded by #496.

Collaborator

jacg commented Jul 28, 2018

I think that this is being superseded by #496.

@jacg jacg closed this Jul 28, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment