Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large dashboard infrastructure restructure #37

Merged
merged 195 commits into from
Oct 3, 2019
Merged
Show file tree
Hide file tree
Changes from 150 commits
Commits
Show all changes
195 commits
Select commit Hold shift + click to select a range
d3c2b38
First pass at implementing new dashboard infra
slyubomirsky Jul 11, 2019
49b7794
Fill in dashboard infra
slyubomirsky Jul 11, 2019
f29d9a5
Don't erase config subdirs between runs (very important)
slyubomirsky Jul 11, 2019
c1d0cbe
update run_dashboard to reflect new structure
slyubomirsky Jul 11, 2019
be5ea75
Remove unnecessary dependency from dashboard_job
slyubomirsky Jul 11, 2019
1cd58ca
Overhaul slack integration
slyubomirsky Jul 12, 2019
a6a78c8
Redundant listcomp
slyubomirsky Jul 12, 2019
a8387d6
Start porting relay opt level experiment to new format
slyubomirsky Jul 12, 2019
4f62d58
Redundant return
slyubomirsky Jul 12, 2019
54effde
Update __init__ to include common
slyubomirsky Jul 12, 2019
05f62b0
Steps after running need access to exp. config
slyubomirsky Jul 12, 2019
4728128
Implement Relay opt level analysis.sh, add analysis and bash utilitie…
slyubomirsky Jul 12, 2019
04583bc
Config should precede input dir
slyubomirsky Jul 13, 2019
8d98aaa
Add visualization script for Relay opt comparison
slyubomirsky Jul 13, 2019
ccc8ce4
Add summarization script for Relay opt level comparison
slyubomirsky Jul 13, 2019
74c2d20
A lot of bugfixes
slyubomirsky Jul 13, 2019
7ec8d11
Execute experiment scripts in their own directory, simplify scripts t…
slyubomirsky Jul 15, 2019
80bef7e
Remove redundant includes from relay_util
slyubomirsky Jul 15, 2019
95e298b
Various bug fixes
slyubomirsky Jul 15, 2019
f3648d1
Port over framework CNN comparison experiment
slyubomirsky Jul 15, 2019
673f02f
Port char RNN to new benchmarking format
slyubomirsky Jul 15, 2019
bc88aa2
Factor out Python script wrapper
slyubomirsky Jul 15, 2019
8cbf1ca
Factor out repeated config validation code, simplify config checking …
slyubomirsky Jul 15, 2019
cf1ae61
Misc fixes and changes
slyubomirsky Jul 15, 2019
701f7e0
Port over treelstm benchmark
slyubomirsky Jul 15, 2019
f208215
Use title field for slack integration, fix TreeLSTM summary title
slyubomirsky Jul 15, 2019
c5c5369
Get rid of old oopsla_benchmarks
slyubomirsky Jul 16, 2019
64013bb
No emacs backups please
slyubomirsky Jul 16, 2019
8a319ea
Factor out repeated summary code
slyubomirsky Jul 16, 2019
d374554
Repair webpage generator
weberlo Jul 18, 2019
00e9c81
Deduplicate dashboard code by using `common`
weberlo Jul 18, 2019
00d76e8
Allow "~" in dashboard config
weberlo Jul 18, 2019
7c2c494
Don't leave slack IDs in the repo (it's sloppy)
slyubomirsky Jul 19, 2019
7874c90
Do CSE pass on `run_dashboard.sh`
weberlo Jul 20, 2019
eb89b2d
Add Relay-to-VTA experiment
weberlo Jul 21, 2019
75f5efe
Add sample config for Relay-to-VTA experiment
weberlo Jul 21, 2019
49f6b06
Uncomment call to delete uncompressed data after compressing
slyubomirsky Jul 23, 2019
9e27d72
Parameterize home dir location in run_dashboard
slyubomirsky Jul 23, 2019
8f39c51
Fix bug in TF script: actually set GPU 0 to be accessible in the GPU …
slyubomirsky Jul 23, 2019
21c7c7f
Permit notify to be empty by default
slyubomirsky Jul 23, 2019
bb748d6
Fix Relay-to-VTA visualize script
weberlo Jul 23, 2019
29ea6fe
Fix more Relay-to-VTA viz
weberlo Jul 23, 2019
10de5e9
Make VTA config modding not look like a steaming pile
weberlo Jul 23, 2019
fdc97fe
Fix VTA config bug in Relay-to-VTA experiment
weberlo Jul 24, 2019
192ada3
Unhackify Relay-to-VTA
weberlo Jul 24, 2019
ab71511
Use `common` module in `slack_integration.py`
weberlo Jul 24, 2019
e1cfb7e
Remove prints
weberlo Jul 24, 2019
1d07c55
Lil bug fix
weberlo Jul 24, 2019
f7ec929
Fix bug in summary construction
slyubomirsky Jul 26, 2019
1d069ed
Add rudimentary training loop eval
slyubomirsky Jul 26, 2019
8491ea0
Add comment in relay_to_vta/run.py
weberlo Jul 26, 2019
9cc9519
Add real Relay-to-VTA summary
weberlo Jul 26, 2019
b438278
Leftover file
slyubomirsky Jul 27, 2019
6907874
Add individual pass comparison benchmark
slyubomirsky Jul 27, 2019
f4d76fc
Add explanatory comment
slyubomirsky Jul 27, 2019
7f82095
Apparently not enough to set CUDA_VISIBLE_DEVICES in Python, somethin…
slyubomirsky Jul 28, 2019
0730fff
Split running Relay TLSTM into multiple scripts to contain AoT resour…
slyubomirsky Jul 29, 2019
5f890ea
Separate out running TF GPU and non-GPU because of environment variab…
slyubomirsky Jul 29, 2019
aa49542
Truncate exceptions in slack message
slyubomirsky Jul 29, 2019
e8b746b
Slightly longer limit for statuses (tweet-length)
slyubomirsky Jul 29, 2019
8e06618
Add plotting utilities
weberlo Jul 29, 2019
41f87ed
Rewrite char rnn experiment to use plot utils
weberlo Jul 29, 2019
5d4dfb8
Fix typo
slyubomirsky Jul 30, 2019
3e9f3bf
Add support for specifying Relay branches in experiment config (use a…
slyubomirsky Jul 30, 2019
36c645f
Update training_loop experiment to use main dashboard's branching fea…
slyubomirsky Jul 30, 2019
0f178a2
Add back missing definition
slyubomirsky Jul 30, 2019
4e8b3d1
Slack integration will list benchmarks for which graphs failed to gen…
slyubomirsky Jul 31, 2019
7335598
Factor out really ugly longitudinal generation code
slyubomirsky Jul 31, 2019
4d6b873
Switch to absolute image heights in webpage gen
weberlo Jul 31, 2019
1052e73
Char RNN's graph should have log scale
slyubomirsky Jul 31, 2019
9dc06aa
Use make_plot to generate the longitudinal comparison
slyubomirsky Jul 31, 2019
c4fe9ed
Used the wrong directory
slyubomirsky Jul 31, 2019
feda9ad
Add multi-bar plots to plot_util
weberlo Aug 1, 2019
d5cf9c2
Remove outdated longitudinal visualization in tlstm
slyubomirsky Aug 1, 2019
eb010cb
Record TVM hash in analyzed data file
slyubomirsky Aug 3, 2019
891001a
Add longitudinal visualizations using last two weeks only
slyubomirsky Aug 3, 2019
c83c1dd
Emphasize individual data points on line graphs
weberlo Aug 2, 2019
211a527
Add Relay-to-VTA comparison graphs
weberlo Aug 3, 2019
e794091
Fix Relay-to-VTA summary
weberlo Aug 5, 2019
905e3d8
Wrap Relay-to-VTA run code in try/except
weberlo Aug 5, 2019
3460d89
Fix longitudinal viz
weberlo Aug 5, 2019
5e27094
Fix Relay-to-VTA viz
weberlo Aug 5, 2019
f02da96
Fix call to Conv2D
slyubomirsky Aug 5, 2019
733b9fb
Add bash wrappers for setting a status file and capturing stderr into…
slyubomirsky Aug 5, 2019
b562586
Corrections to wrappers
slyubomirsky Aug 5, 2019
442b9b6
Wrap fetch and preprocess script
slyubomirsky Aug 5, 2019
a6534dc
Ensure that char rnn experiment pulls language data before starting a…
slyubomirsky Aug 5, 2019
b8ea9ee
Corrections to script wrappers
slyubomirsky Aug 6, 2019
50d99d1
Fix CNN comparison graphs
weberlo Aug 6, 2019
89581f1
Uncomment longitudinal comparisons
weberlo Aug 6, 2019
0b1c2c3
Fix countdown generation for passed deadlines
weberlo Aug 6, 2019
27177a2
Improve bar labeling
weberlo Aug 6, 2019
ca943a4
Don't supply default models, targets, or devices
weberlo Aug 6, 2019
16e9ef2
Catch JSON parsing exceptions and report in experiment precheck
slyubomirsky Aug 7, 2019
bad3422
Improve plot legend
weberlo Aug 7, 2019
4398336
Remove title from RtV plot
weberlo Aug 7, 2019
9c8d52c
Use correct remote and branch when checking out a new TVM
slyubomirsky Aug 8, 2019
fe8127c
Port over Gluon RNN benchmark
slyubomirsky Aug 8, 2019
ac0f0a7
Treat mxnet info as binary
slyubomirsky Aug 8, 2019
820af28
Improve y-axis scales
weberlo Aug 8, 2019
46e705b
Make CNN model names readable
weberlo Aug 8, 2019
3e2895c
Make Char RNN log scale
weberlo Aug 8, 2019
c77f3f6
Repair TreeLSTM benchmark
weberlo Aug 8, 2019
f54aac2
Refactor Y data processing
weberlo Aug 8, 2019
8f44132
Fix longitudinal graph Y scales
weberlo Aug 9, 2019
a3e0685
Use status wrapper in RtV run.sh
weberlo Aug 9, 2019
060a557
Switch over to Matplotlib styles
weberlo Aug 9, 2019
4a656af
Update other frameworks and fix change to VGG16 input format
slyubomirsky Aug 9, 2019
9b5f305
Unnecessary dependency
slyubomirsky Aug 9, 2019
e76e57f
Add post-processed NLP comparison
weberlo Aug 9, 2019
21f9375
Remove unused imports
weberlo Aug 9, 2019
206026f
Make RtV linear scale
weberlo Aug 9, 2019
c47f440
Add trash code to improve bar label positioning
weberlo Aug 9, 2019
2d3b5dd
Fix longitudinal graphs
weberlo Aug 9, 2019
1fc63ef
Add post-processing for opt level comparison and vision graphs
slyubomirsky Aug 10, 2019
4241db3
Use sig fig formatting for bar labels, add configuration option for t…
slyubomirsky Aug 10, 2019
f9563df
Corrections to error wrapper
slyubomirsky Aug 10, 2019
7c7f7c2
Update old visualize scripts
slyubomirsky Aug 10, 2019
da9f1f5
Improve logging safety net
slyubomirsky Aug 12, 2019
9c8ca9b
Desperately try to ensure that treelstm will not OOM on Relay for lar…
slyubomirsky Aug 13, 2019
6fffbdc
Correct use of sequential in relay_util
slyubomirsky Aug 13, 2019
d585d10
Support lists of passes in the pass comparison
slyubomirsky Aug 13, 2019
32ae518
Redundant variables
slyubomirsky Aug 14, 2019
0ae4f48
Add post-processing script for granular pass comparison
slyubomirsky Aug 14, 2019
29ad56e
Redundant part of script
slyubomirsky Aug 14, 2019
357539d
Use build_config to specify which passes to include and exclude for i…
slyubomirsky Aug 14, 2019
a5be4cd
Specify opt_level alongside pass list in pass comparison
slyubomirsky Aug 15, 2019
071d6bb
Try to ensure that baseline case will not fuse
slyubomirsky Aug 15, 2019
e7242ee
Incorrect key
slyubomirsky Aug 15, 2019
ff1ac7f
Permit opt level 4
slyubomirsky Aug 15, 2019
e751e24
Add O4 to visualization
slyubomirsky Aug 15, 2019
37afaab
Use batching for recording times
slyubomirsky Aug 16, 2019
0c6edf5
Coerce batch size to int
slyubomirsky Aug 16, 2019
6a13042
Revert batching
slyubomirsky Aug 16, 2019
52302ae
Update TVM config to have new settings for VTA
slyubomirsky Aug 22, 2019
a95b20a
Plotting improvements
weberlo Aug 22, 2019
676fd75
Fix more viz scripts
weberlo Aug 22, 2019
9052a31
Minor viz tweaks
weberlo Aug 23, 2019
f8bc5b1
First pass on permanent setup system
slyubomirsky Aug 23, 2019
876bfb0
Use setup system for treelstm
slyubomirsky Aug 24, 2019
4b6da9b
Expand wildcards in dashboard config immediately
slyubomirsky Aug 28, 2019
7b873fa
In treelstm's setup, do not rerun if the data is already present
slyubomirsky Aug 28, 2019
63ddcc3
Check if data is present only in run, not setup
slyubomirsky Aug 28, 2019
3790726
Ignore leftover setup directory in tlstm
slyubomirsky Aug 28, 2019
3ce393b
Large subsystem refactor
slyubomirsky Aug 31, 2019
c07afb4
Leftover config.cmake
slyubomirsky Sep 3, 2019
b198ece
Use double quotes for doc comments
slyubomirsky Sep 3, 2019
d3f9700
No actual reason to keep jerry.jpg outside the website subsystem
slyubomirsky Sep 3, 2019
62b71b1
Only read the config if the precheck is valid
slyubomirsky Sep 4, 2019
09fde7b
Add options for setting random seeds
slyubomirsky Sep 4, 2019
63e5653
Centralize dashboard directory logic
slyubomirsky Sep 10, 2019
a8ad6f4
Use setup in gluon RNNs
slyubomirsky Sep 11, 2019
4105ee9
Generate main function parsing instead of doing it everywhere
slyubomirsky Sep 11, 2019
8b50c63
Correctly mark failed setup, force copying
slyubomirsky Sep 11, 2019
8aa344e
Fix broken main methods in gluon RNN
slyubomirsky Sep 11, 2019
fc93436
Fix subsys dir list
slyubomirsky Sep 11, 2019
18c893b
Have the treelstm setup clean up installed files properly
slyubomirsky Sep 11, 2019
47fe456
Use exp title for reporting failed experiments
slyubomirsky Sep 11, 2019
fe597d6
Use correct file locations for website graphs
slyubomirsky Sep 14, 2019
cd22778
Wrong variable in DashboardInfo
slyubomirsky Sep 14, 2019
2c97073
Fix config field names in post_process
slyubomirsky Sep 14, 2019
c3ce81c
Add scoring subsystem and subsystem for posting subsystem results on …
slyubomirsky Sep 14, 2019
a9c204f
Factor out producing timestamp
slyubomirsky Sep 17, 2019
256c8e7
Fix label on Gluon RNN graph
slyubomirsky Sep 17, 2019
7c2d0d6
Accidental reference to removed variable
slyubomirsky Sep 18, 2019
3f78762
Typo in import
slyubomirsky Sep 18, 2019
6d11545
Clear subsystem statuses between runs
slyubomirsky Sep 18, 2019
a825adb
Allow for changing the units of longitudinal graphs
slyubomirsky Sep 18, 2019
465ddf5
Reorganize score implementations, generate score graphs, include scor…
slyubomirsky Sep 18, 2019
69d26bc
Add y scales to score plots
slyubomirsky Sep 19, 2019
a87cac0
Ignore tvm_hash field when traversing analyzed data files
slyubomirsky Sep 19, 2019
0ca5cad
Add high-SD alerting subsystem
slyubomirsky Sep 19, 2019
7816471
Do not keep around old reports for stat alerts
slyubomirsky Sep 19, 2019
7f1ec60
Add more detailed statistical reporting (nasty hack)
slyubomirsky Sep 19, 2019
9aa8dff
Handle exit codes in invoke_main
slyubomirsky Sep 19, 2019
b113341
More (optional) parameters for run dashboard
slyubomirsky Sep 20, 2019
f703cd4
Add templates for commonly repeated steps in experiment steps and por…
slyubomirsky Sep 20, 2019
8da582d
Fix dashboard parameterization
slyubomirsky Sep 20, 2019
749d4f7
Templatize more experiments, improve the templates
slyubomirsky Sep 20, 2019
8285222
Fix incorrect field traversal that was breaking longitudinal graphs, …
slyubomirsky Sep 21, 2019
3f2cb35
Copy raw data when graphing, do not modify raw data in place
slyubomirsky Sep 24, 2019
688fc98
Add rudimentary access to raw data if you want to go mad
slyubomirsky Sep 25, 2019
9a8c31a
Fix incorrect comparison graphs for scores
slyubomirsky Sep 25, 2019
14fd7f7
Correct minor problem in score reporting
slyubomirsky Sep 25, 2019
091b4bf
Desperately try copying again to see if there's an in-place modificat…
slyubomirsky Sep 25, 2019
cbf3844
Copy again for good measure
slyubomirsky Sep 25, 2019
8373d97
Make proper deep copy of raw data when graphing
slyubomirsky Sep 26, 2019
5a21342
Left a print in by accident
slyubomirsky Sep 26, 2019
44e2c55
Set a sliding window for stat alerts
slyubomirsky Sep 26, 2019
fe82bc2
Set all random seeds that we care about
slyubomirsky Sep 27, 2019
898fdbd
Overhaul training loop to use SMLL verison
slyubomirsky Sep 27, 2019
211a068
Improve plot_util docs and fix bug with ints
weberlo Sep 27, 2019
3ab4dd3
Add /usr/local/bin to PATH for the cron environment
slyubomirsky Sep 27, 2019
1ac3056
Don't fetch from origin when rebuilding between experiments
slyubomirsky Sep 27, 2019
cdfa9fa
Update readme
slyubomirsky Oct 3, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ __pycache__/
# Java class files
*.class

# Emacs backups
*~

# Distribution / packaging
.Python
build/
Expand Down Expand Up @@ -120,11 +123,21 @@ source.cc
*.DS_Store

# treelstm data
oopsla_benchmarks/pytorch/rnn/tlstm/data/*
oopsla_benchmarks/pytorch/rnn/tlstm/lib/stanford-tagger/*
oopsla_benchmarks/pytorch/rnn/tlstm/lib/stanford-parser/*
experiments/treelstm/pt_tlstm/data/*
experiments/treelstm/pt_tlstm/lib/stanford-tagger/*
experiments/treelstm/pt_tlstm/lib/stanford-parser/*
experiments/treelstm/pt_tlstm/lib/stanford-postagger-2015-01-30/*
experiments/treelstm/pt_tlstm/lib/stanford-postagger**
experiments/treelstm/setup/*

# pulled-in resources for training loop
experiments/training_loop/beacon

# data and graphs
*.csv
*.png
*.zip

# leftover files from copying setups
.last_setup
status.json
207 changes: 0 additions & 207 deletions analyze.py

This file was deleted.

9 changes: 9 additions & 0 deletions dashboard/config.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -133,3 +133,12 @@ set(USE_SORT OFF)

# Build ANTLR parser for Relay text format
set(USE_ANTLR OFF)

# Whether to build fast VTA simulator driver
set(USE_VTA_FSIM ON)

# Whether to build cycle-accurate VTA simulator driver
set(USE_VTA_TSIM ON)

# Whether to build VTA FPGA driver (device side only)
set(USE_VTA_FPGA OFF)
Loading