Resnet dense layers FAIL unless preceded by e.g. gaussian test

[steveri]

Executive summary: I'm not sure we are properly testing dense resnet layers.

Details:

I'm not sure what to do with this information (yet). And I have not done the work to thoroughly, under controlled conditions, verify what's really happening (yet). But here's what it looks like:

I tried using the "aha regress" command to run 'conv1' on its own, as a dense test, and it failed.

But! If I include a non-layer test e.g. gaussian in the test suite, e.g. "gaussian" followed by "conv1", then both tests pass.

I run the tests with the garnet daemon turned ON, which means that the "conv1" test (re)uses the verilog that was built for the "gaussian" test, in the case where both tests pass. I think this is relevant. But I'm not sure what it means, since both are dense tests (I think?) and both should be using the same verilog anyway (right?)

Te next thing I would probably try: What happens if we run "gaussian" + "conv1" with daemon OFF? I'm guessing that conv1 fails.

Until I have better information, I guess I will file this as both a "garnet" issue and an "aha" issue...

I will include @kalhankoul96 as an assignee, because I think he'll be interested, and because he can remove himself and/or add more assignees if there's anyone else that might obviously want to get notifications about this...

Aha issue is here: StanfordAHA/aha#1897