Add new feature of filtering rows by status. Minor changes like displ…

[zkdtc]

Add new feature of filtering rows by status. Minor changes like display SPECPROD in the heading, not including tonight for daily processing if using --nights

[akremin]

Thanks Kai. The code itself looked good, but when I went to test it on cascades I got an error:
Running: desi_proc_dashboard --output-dir=${WWWLOC}/ --output-name=dashboard.html --start-night=20201214 --end-night=20210224

Output:

Searching /global/cfs/cdirs/desi/spectro/redux/cascades for  ['20210224' '20210223' '20210222' '20210221' '20210220' '20210219'
 '20210218' '20210217' '20210216' '20210215' '20210214' '20210213'
 '20210212' '20210211' '20210210' '20210209' '20210208' '20210207'
 '20210206' '20210205' '20210204' '20210203' '20210202' '20210201'
 '20210131' '20210130' '20210129' '20210128' '20210127' '20210125'
 '20210124' '20210123' '20210122' '20210121' '20210119' '20210118'
 '20210117' '20210116' '20210115' '20210114' '20210113' '20210112'
 '20210111' '20210110' '20210109' '20210108' '20210107' '20210106'
 '20210105' '20210104' '20210103' '20210102' '20210101' '20201231'
 '20201230' '20201229' '20201228' '20201227' '20201226' '20201223'
 '20201222' '20201221' '20201220' '20201219' '20201218' '20201217'
 '20201216' '20201215' '20201214']
Month: 202102, nights: ['20210224', '20210223', '20210222', '20210221', '20210220', '20210219', '20210218', '20210217', '20210216', '20210215', '20210214', '20210213', '20210212', '20210211', '20210210', '20210209', '20210208', '20210207', '20210206', '20210205', '20210204', '20210203', '20210202', '20210201']
Traceback (most recent call last):
  File "/global/homes/k/kremin/workspace/desi_proc_dashboard2/desispec/bin/desi_proc_dashboard", line 17, in <module>
    main(args)
  File "/global/homes/k/kremin/workspace/desi_proc_dashboard2/desispec/py/desispec/desi_proc_dashboard.py", line 264, in main
    nightly_tables.append(nightly_table(night,skipd_expids,show_null=args.show_null,use_short_sci=args.include_short_scis))
  File "/global/homes/k/kremin/workspace/desi_proc_dashboard2/desispec/py/desispec/desi_proc_dashboard.py", line 318, in nightly_table
    table_row = _table_row(row_info[1:],idlabel=row_info[0])
  File "/global/homes/k/kremin/workspace/desi_proc_dashboard2/desispec/py/desispec/desi_proc_dashboard.py", line 803, in _table_row
    if elements[14]=='unprocessed':
IndexError: list index out of range

In digging further the code fails whenever I give it a range in which a night doesn't have an exposure table (e.g. the nights of Dec 24-31 2020, Jan 4 2021 etc.)

[sbailey]

Anthony identified the more important issue (crashing), but another minor point: let's keep this as the original method of specifying the output-name including the extension so that we don't arbitrarily break existing cronjobs, notes, etc. that expect to specify --output-name dashboard.html

[sbailey]

Let's change this back to not auto-adding the extension, to keep it the same as it was before.

[zkdtc]

Thanks for pointing out the errors, @akremin @sbailey
The crashing is because when no exptable and processing tables are present, the code uses the traditional way of scanning files but does not add status to the row. Now I assume if no exptable or processing tables are there, then they are unprocessed, which seems logical. Let me know if this should be changed.

[akremin]

That sounds logical to me too. Thank you. I'll check again and give a review

[akremin]

I was able to run on the full cascades and it works well. The three options all seem to work and I can toggle between them.