Not exactly sure how to make this work #10

Closed
mertonium opened this Issue Mar 7, 2014 · 5 comments

Comments

Projects
None yet
3 participants

This gem sounds great and I want to use it. I'm just not sure how.

The documentation seems to have most of the parts, but it lacks an overall explanation of work flow.

I've added the gem to my gemfile, made the tmp/coverband_baseline.json file, added the rake tasks and the middleware. Then I wanted to test it locally, but the README kind of stops there.

The coverband_baseline rake task seemed like a good place to start, so I ran it (with --trace). A bunch of json spat out and no errors, so hurray. I did notice, however, that the filenames listed in the json only included my initializers.

With the baseline taken, I assumed the only other thing to do was run the coverband rake task, so I did. When it was finished, my browser popped open to show me a lovely simplecov page. The only problem was that it only listed my initializer files.

Could someone explain to me 1) what the ideal coverband workflow is (add all the stuff from the readme, deploy it, run the baseline, then the coverband tasks?) and 2) any reason why the rest of my production code was missed out?

I'd be happy to submit a PR with more setup instructions in the README once I get my head wrapped around these concepts.

Owner

danmayer commented Mar 8, 2014

Good call, the directions were a bit incomplete. Basically after you configure things you need to start your development server and make sure cover band is loaded. I also recommend first testing in development with coverage set at 100%, with verbose setting true, and logging enabled.

I updated the usage section of the README, let me know if this helps: https://github.com/danmayer/coverband#usage

danmayer closed this Mar 8, 2014

sdhull commented Oct 28, 2014

I'm just curious what is the purpose of the baseline? I checked the definition of parse_baseline and it defaults to an empty hash if it can't find the tmp/coverband_baseline.json file. How important is the baseline and what happens if the baseline does just default to an empty hash?

Owner

danmayer commented Oct 30, 2014

hmm Sorry @sdhull that the purpose isn't clear enough. I will try to work on improving the docs some more. Basically the way coverband works is to record lines of code executed with set_trace_func, which this works well it doesn't give the proper output for many lines of code, for example most end lines that close blocks aren't ever 'executed'. It also doesn't really cover method definition, require lines, and includes... All of these get the proper output using the standard libraries Coverage, which is what we are trying to emulate with set_trace_func. Since Coverage can't be used with sampling during production runtime, the best compromise is to record a baseline first with Coverage just recording how it sees the code executed while loading. Then merging the lines executed during runtime later.

So it breaks down to two pieces.

  1. baseline is a one time static recording of code executed when the app is booted.
  2. runtime coverage is recorded to redis with the coverband rack-middleware

When we generate the code coverage report we merge the two together.

sdhull commented Oct 30, 2014

Awesome, that helps a lot! So it's not important to run generate the baseline prior to running in production, only prior to generating the coverage with the rake task. That's the "usage" bit I was unsure about.

As for coverage, I wonder what the performance impact would be for running at 100%. If my app had 10 appservers, I could deploy with coverage running on one of them for 10% sample rate 😉

Owner

danmayer commented Nov 1, 2014

yep you can run it any time to have the baseline to merge when making the report. Glad that helped.

@sdhull so running on a single server as a way of sample is a good idea. Coverage does have pretty good perf, but there are two other issues.

  1. To gather the data from coverage you need to stop it... end the coverage run, since it can't be turned back on... when do we choose to gather and collect the data... After running for an hour? Try to trap signals? midnight?
  2. Coverage mostly related to trying to stop and start also seemed to eventually have segfaults. So running it full time on the server might run into some increased segfaults.

Anyways, could be fun to play around with and see if it works out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment