Generating a static HTML snapshot/report of coverage after having run coverband in production for a period

[odinhb]

I've been running Coverband in production for about a month and it has produced for me a redis database file which I can download and inspect locally without worrying about deployments changing or destroying any data.

The problem is that as we start deploying new code, coverage information is 'destroyed' (inaccessible) for the updated file (I'm using HashRedisStore), so I'd like to take a kind of 'snapshot' at this moment in time, but it is rather cumbersome to use this database as-is:

# [some commands to locate and download the database from prod]
# term 1
redis-server --dbfilename some-database.rdb --port 6381
# term 2
ENABLE_COVERBAND=true rake coverband:coverage_server
# then open a browser at localhost:6381

I'm using SimpleCov for code coverage when running tests, and I noticed that the product of this tool is an almost self-contained html file (save for some static assets), which is really easy to open (just drop it into your browser) and more performant (it contains all the source code inline in the html from the get-go).

In order to get a similar, permanent snapshot of my codebase I created a really hacky script (which doesn't work without rails 😅) which can archive the (important bits of) html output from rake coverband:coverage_server.

Something I noticed while writing and running my script is that it takes an abnormally long time to fetch the details for individual files. This is another motivation for a simple set of html files: it feels much snappier to navigate.

But we noticed it while trying out the web UI, too, so I took a look at redis with redis-cli -p 6381 monitor and it looks like every time I click an individual file in the coverage_server web-UI, it loads the coverage data for every source file in the project. I think there's a missing optimization there.

I think Coverband ought to provide a simpler way to get a static, human-friendly snapshot of the data, and considering the fact that my hacky export script produces html that can be handed to a webserver and served statically, I know that this goal is not mutually exclusive with providing a small rack app that serves the most recent snapshot on production.

[kbaum]

I think there's a missing optimization there.

Agreed. I think the reporting code is agnostic of the coverage adapter. With HashRedisStore the data for each file is stored under a separate key while with the default RedisStore everything is stored under one key. We would have to optimize the reporting to be able to take advantage of the way HashRedisStore stores the data. We have discussed the idea of removing the RedisStore in favor of HashRedisStore. This would make an optimization like you suggest easier but there is some concern that HashRedisStore puts too heavy of a performance impact on the redis instance. Curious what kind of performance impact you see using HashRedisStore?

I think Coverband ought to provide a simpler way to get a static, human-friendly snapshot of the data,

Coverband used to come with a rake task that just generated static html like which I think is what you want. Somewhere along the line we removed it in favor of the small rack app. I can't remember exactly why we removed the static html but my guess is that we just wanted to delete code to simplify the project :). Perhaps we should consider bringing back the html generation but I'm not sure how much effort that would be today.

Another option is the coverband.io service which I think would meet your use case as it supports coverage history.

[odinhb]

Curious what kind of performance impact you see using HashRedisStore?

We haven't noticed any performance impact using HashRedisStore. Our deployment is not that big, so it probably won't be a problem for us.
I went with HashRedisStore w/ very little concern for performance. To me it seems essential that coverage drift is eliminated, and HashRedisStore is the only adapter that counters this w/ checksumming. Otherwise we would probably have gone w/ FileStore. We also wanted to use the dead_methods task (not supported by FileStore).

TLDR; I picked HashRedisStore purely to get all the little features missing from other adapters, with little concern for performance.

With HashRedisStore the data for each file is stored under a separate key while with the default RedisStore everything is stored under one key.

The coverage adapters can (as far as I can see) stay agnostic, but would provide a method for the webUI's file details which returns coverage data for just the one file at a time. In the "old-school" adapters this would fetch all the data like before and then filter it in memory (maybe this means moving some filtering logic into the adapter), but HashRedisStore could just fetch a single key.

Perhaps we should consider bringing back the html generation but I'm not sure how much effort that would be today.

It looks kind of non-trivial to me as well, as the rack app that runs in rake coverband:coverage_server does a lot of work which is not trivial to replace statically (as my script demonstrates by being dirty and long), but it should be possible to generate the html report differently.

I imagine you could have a script which generates the static html, and then the rack app would present a button which runs said script. If the script output exists in PROJECT/coverband or wherever, the rack app would serve it as is. Then I could just choose whether I'd like to run the script and then open index.html through the rack app or using file:///.

[kbaum]

The coverage adapters can (as far as I can see) stay agnostic, but would provide a method for the webUI's file details which returns coverage data for just the one file at a time. In the "old-school" adapters this would fetch all the data like before and then filter it in memory (maybe this means moving some filtering logic into the adapter), but HashRedisStore could just fetch a single key.

This makes perfect sense.

I imagine you could have a script which generates the static html, and then the rack app would present a button which runs said script. If the script output exists in PROJECT/coverband or wherever, the rack app would serve it as is. Then I could just choose whether I'd like to run the script and then open index.html through the rack app or using file:///.

I would rather we brought back something that generated the report statically and then the rack app leveraged that same logic but made the appropriate modifications before rendering to the client.

[danmayer]

yeah, we removed the static HTML to remove and reduce some of the complexity and believed the rack app to be a better experience. We could try adding some of that back, but there hasn't been too much demand. If we want to add it back, I believe the issue was we didn't have sufficient tests around the static file and it kept breaking or diverging from the rack app and we didn't notice until getting bug reports... As I no longer use the static page in any of the apps I run, nor do I connect to a remote redis as all the production redis environments are behind a firewall.

[anit]

Yes. This can be an amazing feature. We need a way to extract static reports after the coverband process. Helps a lot in reporting.

[FlorianASchroeder]

in the meantime you could export the HTMLReport using your own rake task

report = Coverband::Reporters::HTMLReport.new(Coverband.configuration.store, {base_path: './coverage', filename: 'index.html'}).report
File.open('coverage/index.html','w') do |f|
  f.write(report)
end

or as JSONReport:

report = Coverband::Reporters::JSONReport.new(Coverband.configuration.store).report
File.open('coverage/coverage.json','w') do |f|
  f.write(report)
end

[danmayer]

OK, finally going to get back to this one and close it out. @FlorianASchroeder thanks for the tip, I basically copied your method for the JSON report and added it to a rake task... the html report doesn't really work unless you are running the server locally at the time you load the file... otherwise it will fail to find things... so I took a bit of a different approach.

I don't want to have to go back to a fully static report generator there were a number of reasons that wasn't great including how long it could take to generate for larger projects... so it could timeout while generating. The new rack app has a number of advantages and flexibility for the project going forward...

That being said, Coverband can still output a Simplecov compatible report... so the new static HTML report just passes our data to the simplecov-html reporter which will build a fully static report. This way I won't have to maintain a static report, so long as Coverband and SimpleCov continue to maintain a data format that is compatible... and we have for a decade, so that feels OK.

Thanks everyone who kept up with this ticket or added emojis, comments, or feedback.

Anyways, new tasks will be in the next release:

• bundle exec rake coverband:coverage_json
• bundle exec rake coverband:coverage_html