Saving the HTML from coverage runs to provide samples

[rseac]

Curious whether the following is a good idea to provide examples: the coverage runs jekyll-scholar and creates the HTML files. Is it possible to use the same infrastructure to create sample HTML files and keep them instead of deleting them as specified in the support files?

Ideally, one could create their sample by writing a feature test. Furthermore, each of the feature tests could serve as rendered examples that people can look at.

[inukshuk]

Something along those lines would be awesome. I have no concrete plans, but is painfully obvious that
the README is too long and hard to maintain; not to mention parts of it are likely out of date. There is a lot of information scattered in issue threads and in the tests that is hard to find for new users. Since jekyll-scholar is basically just a collection of small plugins and helper functions, most users would probably prefer a catalogue of examples to an old school manual / readme page. So I think you're totally right: if we could set up a set of examples which can be viewed and tested automatically (to catch regressions mostly) that would be great.

We're setting up a temporary folder where all the cucumber features are rendered. We could probably add a hook which saves those files and uploads them somewhere at the end of a run on Travis CI, so that might be a worthwhile idea.

An easier approach might be to curate a few examples and generate a site if all tests are green (that may be easier, but I haven't given it much thought).

[rseac]

Here's a starting point that may help in making this a bit more concrete (some tests fail). I do not know cucumber at all so I would need your help in making this work correctly, but hopefully it gives you an idea.

We could push the generated HTML onto github pages for jekyll-scholar.
!cat features/support/hooks.rb

Before do
  FileUtils.rm_rf(TEST_DIR) if File.exist?(TEST_DIR)
  FileUtils.mkdir_p(TEST_DIR)
  Dir.chdir(TEST_DIR)
end

After do |scenario|
 merge_outputs(scenario)
 FileUtils.rm_rf(TEST_DIR) if File.exist?(TEST_DIR)
end


def merge_outputs(scenario)
   # Merge into this file
   file = File.open("/tmp/test.html", "a")

   file <<  "<h1>Title: #{scenario.title} </h1>"
   # Check if there is an HTML file.

   Dir.entries(TEST_DIR).each do |name|
      if File.extname(name) == ".html"
         # Source file formatting
         file << "<pre>" + File.open(name).read + "</pre>"
         # Processed
         file << "<br /> <h2>Jekyll-scholarized</h2>"
         file << File.open("_site/" + name, "r").read
         file << "<br />"
      end
   end

   file.close()
end

[inukshuk]

It's certainly a good idea, but I think it may become too complicated to support all tests (and also ensure that all tests are meaningful examples).

However, I think something that we could do is set up a jekyll site with the readme and examples (it does not even need to be tied to this repository). The site's repository could include Travis CI hooks which just build the page and push it back to GitHub pages. That would make it very easy for anyone to contribute examples using PRs. What do you think?

[rseac]

Fine by me.

[rseac]

Something like this: jekyll-scholar-examples?

[inukshuk]

Exactly. What we'd need to add is a hook for Travis CI where we compile the site and then push it back to GitHub: i.e., continuous deployment for the examples (how to set something like this up is useful example by itself). If you're willing to work on this we could set up a GitHub org for jekyll-scholar and move this repository and examples there (also extras?).

[rseac]

Used what you had for your CI setup. Please see now.

What about publishing the rendered examples on github pages? I don't know how to do this, do you?

[inukshuk]

Yes, we have to do something like this:

Setup GitHub pages for the examples repo (in a separate branch, typically the branch is called gh-pages I think). This branch contains the generated page (basically the _site folder). Then we need to set up Travis CI for the master branch with a CI script that does this:

1. Install gems (that's the install script in .travis.yml)
2. Build the site to _site (that's script in travis.yml)

If the commit is from a PR then that's it. Otherwise, it should execute the deployment part (this is after_success in travis.yml, see here):

1. Check out the gh-pages branch to a temporary location. This is the 'output folder'.
2. Clears the contents of that folder.
3. Copies the contents of _site generated by the build script to the output folder.
4. Commit and push the output folder back to the gh-pages branch (we will need to set up encrypted credentials for that).

[rseac]

Should be setup now. .travis.yml.

[inukshuk]

Good work! I wasn't aware that this setup is now supported directly by Travis CI via the pages provider -- that's pretty awesome.

Anyway, this is really great. We can start porting over examples from the Readme and it will be really helpful to many users. Do you want the repo to stay where it is or shall we set up a GitHub organization and move all the repositories there?

@IgnoredAmbience what do you think of the idea?

[rseac]

@inukshuk Either way is fine.

[rseac]

Quick update. Each example within example/ is now independent with its own _config.yml file, etc. This is to allow good examples such as the one provided by @IgnoredAmbience in one of the issue discussions to (hopefully) render as well.

[IgnoredAmbience]

Apologies, didn't see this at the time. Seems like a good idea to me. Still would also be nice if we can automatically add the test cases to a subdirectory of the site too. I'm tempted to try this when I have some free time. (Currently taken on too much to do).

[IgnoredAmbience]

If anything, having them output somewhere less ephemeral will make it easier to configure a mode to debug failing test cases. I currently have to hack around with the cucumber support code to turn off the temp dir deletion, which is mildly annoying.

[rseac]

@IgnoredAmbience You could turn it off by changing this:

jekyll-scholar/features/support/hooks.rb

Line 8 in d11c063

FileUtils.rm_rf(TEST_DIR) if File.exist?(TEST_DIR)

[IgnoredAmbience]

@rseac Yes, that's what I currently do. Remembering to not accidentally commit the change is hard, etc.

Making this configurable would be a code change made at the same time as placing such test cases into an (test case section of) an examples repo.

[inukshuk]

What about adding an environment variable for this? We'd just have to add a single line and you could set it by default and not have to think of it again (e.g., SCHOLAR_KEEP_TMPDIR).