Reformat model information and add link to full channel list #78

macedo22 · 2018-03-01T00:32:26Z

No description provided.

coveralls · 2018-03-01T00:48:48Z

Coverage remained the same at 7.957% when pulling 6dd7498 on macedo22:small-changes into 1a9c842 on ligovirgo:master.

coveralls · 2018-03-01T00:48:48Z

Coverage remained the same at 7.957% when pulling 2ba4227 on macedo22:small-changes into b55aa97 on ligovirgo:master.

duncanmmacleod · 2018-03-13T16:08:48Z

bin/gwdetchar-lasso-correlation

+except(IOError, IndexError):
+    numpy.savetxt(channelsFile, channels, fmt="%s")
+except RuntimeError as e:
+    if 'latex' in str(e).lower():


@macedo22, why would latex get involved here?

@duncanmmacleod Not sure. I am not too familiar with Latex and because this exception handling was used for saving all of the plots, I assumed it would go here too. I figured the handling was to account for characters in strings that Latex would get mad at, but it sounds like what you're saying is a Latex exception wouldn't be raised from saving channel names to a text file, as it could for images. I will remove the if/else conditional here and just have
except RuntimeError:
raise

there should be no need for any exception handling here at all, so I recommend you remove all of it, unless you understand a configuration where the file writing should be allowed to fail.

macedo22 · 2018-05-29T22:30:06Z

Hi @duncanmmacleod, I removed the error handling you requested as well as a few other things.

process_channel now only iterates through the channels with a non-zero lasso coefficient
The boolean clustering argument was defaulting to True before, but it is now fixed and works as expected.
Non-zero channels and their lasso coefficients are now stored in a dictionary and accessed by key instead of indexed in an array.

duncanmmacleod · 2018-06-07T18:54:35Z

bin/gwdetchar-lasso-correlation

@@ -99,20 +100,15 @@ parser.add_argument('-J', '--nproc-plot', type=int, default=None,
 parser.add_argument('-o', '--output-dir', default=os.curdir,
                    help='output directory for plots')
 parser.add_argument('-f', '--channel-file', type=os.path.abspath,
-                    help='path for channel file')
+                    help='path for channel file', required=True)


@macedo22, I'm not a fan of required optional arguments, since that makes no sense? Can you have a think about changing this to a positional argument if it really is required?

@duncanmmacleod Yes, I can! I can just remove it to keep options open, as you say

duncanmmacleod · 2018-06-07T18:57:36Z

bin/gwdetchar-lasso-correlation

@@ -128,7 +124,7 @@ psig.add_argument('-x', '--filter-padding', type=float, default=3.,
 lsig = parser.add_argument_group('LASSO options')
 lsig.add_argument('-a', '--alpha', default=None, type=float,
                  help='alpha parameter for LASSO fit')
-lsig.add_argument('-C', '--cluster', default=True, type=bool,
+lsig.add_argument('-C', '--cluster', default='True', type=str,


I don't understand this change, can you explain?

@duncanmmacleod As I discovered from troubleshooting and reading more online, for some reason, boolean arguments will always be True whether you pass in True or False. There are a few different ways to work around this issue, but I found this the simplest and most readable.

I believe it has something to do with the nature of python or parsing. It will treat any string passed as a boolean as True

in this case you want to use action='store_true' and just give the argument with a value, e.g.

lsig.add_argument('-C', '--cluster', action='store_true', help='generate clustered channel plots')

the user should then pass just --cluster to enable the clustering.

@duncanmmacleod I'm happy to change this, but our overall usage preference has been to always do clustering by default, and only turn it off if in special cases (e.g. when troubleshooting the code overall to fix bugs). If we would like to keep it as doing clustering by default without the user passing anything, is there another way you would like me to rewrite it? If you still feel this new way you proposed is best, I will make the switch.

@macedo22, you can use the opposite logic and provide something like:

lsig.add_argument('-C', '--no-cluster', action='store_true', default=False, help='do not generate clustered channel plots')

I believe that one of those options using action='store_true' is better than having the user manually pass true or false on the command-line.

@duncanmmacleod Sounds good to me. I swapped this in and tested it, so it should be good to go with this most recent commit 6dd7498.

duncanmmacleod · 2018-06-07T18:58:07Z

bin/gwdetchar-lasso-correlation

 auxdata = gooddata
+flattab=Table(data = (numpy.asarray(flatdata.keys()),), names = ('Channels',))


@macedo22, please use pep8 white-spacing standards throughout.

@duncanmmacleod will do!

macedo22 · 2018-06-12T01:18:42Z

@duncanmmacleod I went ahead and put in your edits. In addition, I have swapped out LassoCV as the default process to use to find the best alpha value when none is given. Jeff and Marissa have been experimenting with information criterion, and it seems to be more useful and informative for our purposes. I replaced LassoCV with using information criterion to find the best alpha to use if none is given.

duncanmmacleod · 2018-06-12T08:46:55Z

bin/gwdetchar-lasso-correlation

+write_param('Model', 'LASSO')
+write_param('Non-zero coefficients', '%d' % numpy.count_nonzero(model.coef_))
+write_param('Alpha', '%g' % usedalpha)
+write_param('Zero coefficients','%d (%s)' % (len(zeroedtab), "<a href= %s target='_blank'>zeroed channel list</a>" % (zerofile)))
 page.p('<br /><br />%s' % df.to_html(index=True))
 page.div.close()#model-info


as per pep8, please leave two spaces before inline comments:

page.div.close() # model-info

macedo22 · 2018-07-09T21:16:01Z

Hi @duncanmmacleod, I did an overhaul of the code in an attempt to make it more robust. The changes include:
-the large summary plots at the top are generated before the individual auxiliary channel plots
-instead of using pyplot to edit and set features of the plots, the Figure objects themselves are used
-we kept getting TimeoutErrors and LOCKERRORs, and the added config_mpl() function (https://github.com/macedo22/gwdetchar/blob/f0b615b05f456e3efc437ced6d990a3065dc3a8f/bin/gwdetchar-lasso-correlation#L94) is meant to reconfigure matplotlib to store each individual plot's tex cache in its own location, instead of them all sharing the same tex cache folder. This is to avoid two or more plots simultaneously trying to acquire lock with the shared tex cache folder. Since this was added, we are not encountering any of the above errors anymore.

macedo22 · 2018-07-13T05:08:31Z

An updated example page output: https://ldas-jobs.ligo-la.caltech.edu/~alexandra.macedo/detchar/LASSO/test/O2/L1/20170301/1172363755-1172375019/

duncanmmacleod

LGTM

duncanmmacleod · 2018-06-13T17:28:27Z

bin/gwdetchar-lasso-correlation

    return plot7, plot7_legend


-if args.cluster is True:
+if args.no_cluster is False:


@macedo22, by convention its normally better to use if not args.no_cluster for this (and similarly just if args.cluster for truth testing).

duncanmmacleod · 2018-07-18T16:10:39Z

@macedo22, this looks fine to me, the CI failure is from somewhere else and is related to backwards-incompatible gwpy changes that haven't been addressed in gwdetchar yet, I'll get to it soon.

macedo22 · 2018-07-23T17:02:46Z

@duncanmmacleod Great! It seemed like the issue was in gwdetchar-scattering, but let me know if there ends up being any unanticipated compatibility issues with gwdetchar-lasso-correlation and I can address them on this branch before merge

duncanmmacleod · 2018-07-23T18:15:05Z

@macedo22, I fixed that failure in #87, so if you can rebase your feature branch against the upstream master and force push, you should be able to get past that.

coveralls · 2018-07-25T01:52:03Z

Pull Request Test Coverage Report for Build 236

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 39.372%

Totals
Change from base Build 235:	0.0%
Covered Lines:	426
Relevant Lines:	1082

💛 - Coveralls

macedo22 · 2018-07-25T02:15:06Z

@duncanmmacleod Done and done!

duncanmmacleod reviewed Mar 13, 2018

View reviewed changes

macedo22 force-pushed the small-changes branch from 6adc2e6 to cb8a5b8 Compare May 31, 2018 17:30

duncanmmacleod reviewed Jun 7, 2018

View reviewed changes

duncanmmacleod reviewed Jun 12, 2018

View reviewed changes

macedo22 force-pushed the small-changes branch from 56d11d5 to 6dd7498 Compare June 13, 2018 17:27

duncanmmacleod self-requested a review July 18, 2018 16:09

duncanmmacleod approved these changes Jul 18, 2018

View reviewed changes

duncanmmacleod self-assigned this Jul 18, 2018

duncanmmacleod added the enhancement label Jul 18, 2018

macedo22 force-pushed the small-changes branch from 4f3218e to f5077ea Compare July 25, 2018 01:47

Alexandra Lynn Macedo and others added 7 commits July 30, 2018 13:41

Reformat model information and add link to full channel list

b1fa68f

Process non-zero channels and reformat data

5ef0f4b

Can do other channels than range plus band pass capability

31aa72b

Fix scatter plot y range

1f9c4f0

Fix scatter plot lower y bound

b08236b

Fix cluster percentage completion calculation

bdf78ef

Report sigma and cluster coefficient threshold on webpage output

0b7fbcb

Alexandra Lynn Macedo and others added 13 commits July 30, 2018 13:41

Use information criterion when no alpha given

3531cfb

Restyle with PEP 8 formatting

f4ddb00

Change clustering argument boolean implementation

142e180

Set plot x-axes to be in hours

d74b354

Change naming convention of saved files

cd07751

Info criterion can't pick 0 channels

42c560f

Fix spacing typos

4169ddd

Handle Inf/NaN ValueErrors

ceaba62

Fix clustering and resize images

7d2bebf

Move plots, change calls to and reconfigure matplotlib

b19f84d

Limit the number of clustered channels plotted per top channel

3bafe67

Make results table less ugly

45b36b3

De-acronym 'LASSO' to lasso

2d047f7

macedo22 force-pushed the small-changes branch from dde1409 to 2d047f7 Compare July 30, 2018 18:42

duncanmmacleod merged commit 52af2f0 into gwdetchar:master Jul 31, 2018

macedo22 deleted the small-changes branch August 7, 2018 16:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reformat model information and add link to full channel list #78

Reformat model information and add link to full channel list #78

macedo22 commented Mar 1, 2018

coveralls commented Mar 1, 2018 •

edited

coveralls commented Mar 1, 2018

duncanmmacleod Mar 13, 2018

macedo22 Mar 14, 2018 •

edited

duncanmmacleod Apr 9, 2018

macedo22 commented May 29, 2018

duncanmmacleod Jun 7, 2018

macedo22 Jun 7, 2018 •

edited

duncanmmacleod Jun 7, 2018

macedo22 Jun 7, 2018

macedo22 Jun 7, 2018

duncanmmacleod Jun 12, 2018

macedo22 Jun 12, 2018

duncanmmacleod Jun 13, 2018

macedo22 Jun 13, 2018 •

edited

duncanmmacleod Jun 7, 2018

macedo22 Jun 7, 2018

macedo22 commented Jun 12, 2018

duncanmmacleod Jun 12, 2018

macedo22 commented Jul 9, 2018 •

edited

macedo22 commented Jul 13, 2018

duncanmmacleod left a comment

duncanmmacleod Jun 13, 2018

duncanmmacleod commented Jul 18, 2018

macedo22 commented Jul 23, 2018

duncanmmacleod commented Jul 23, 2018

coveralls commented Jul 25, 2018 •

edited

macedo22 commented Jul 25, 2018

		auxdata = gooddata
		flattab=Table(data = (numpy.asarray(flatdata.keys()),), names = ('Channels',))

Reformat model information and add link to full channel list #78

Reformat model information and add link to full channel list #78

Conversation

macedo22 commented Mar 1, 2018

coveralls commented Mar 1, 2018 • edited

coveralls commented Mar 1, 2018

Choose a reason for hiding this comment

macedo22 Mar 14, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

macedo22 commented May 29, 2018

Choose a reason for hiding this comment

macedo22 Jun 7, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

macedo22 Jun 13, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

macedo22 commented Jun 12, 2018

Choose a reason for hiding this comment

macedo22 commented Jul 9, 2018 • edited

macedo22 commented Jul 13, 2018

duncanmmacleod left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duncanmmacleod commented Jul 18, 2018

macedo22 commented Jul 23, 2018

duncanmmacleod commented Jul 23, 2018

coveralls commented Jul 25, 2018 • edited

Pull Request Test Coverage Report for Build 236

💛 - Coveralls

macedo22 commented Jul 25, 2018

coveralls commented Mar 1, 2018 •

edited

macedo22 Mar 14, 2018 •

edited

macedo22 Jun 7, 2018 •

edited

macedo22 Jun 13, 2018 •

edited

macedo22 commented Jul 9, 2018 •

edited

coveralls commented Jul 25, 2018 •

edited