Breaking up this repo

[gvwilson]

This issue is for discussion of our plans to break this repo into lesson-sized pieces. Please use it for comments on lesson templates, mechanics of importing material via Git, etc.

[wking]

On Sun, Sep 28, 2014 at 12:43:02PM -0700, Greg Wilson wrote:

This issue is for discussion of our plans to break this repo into
lesson-sized pieces.

I'm in favor of this breakup, whatever it looks like (obviously: #102
;). However, I don't really see a need to simultaneously adjust the
default procedure for consuming swcarpentry/bc. In your plan
(swcarpentry/DEPRECATED-site#610):

Almost all workshops will be able to set themselves up by cloning
just that repository, which will (probably) contain only a couple of
dozen files, then point at rendered lessons on the Software
Carpentry website instead of having their own (redundant) copies.

I think redundant copies are a good thing, because they avoid link
rot (or a maintenance burden on centrally hosted content, see #488 and
the resultant fallout in #496 and #503). If you branch off the
central workshop repository, I'd rather you had as few external
dependencies as possible. For example, it would not be fun to show up
to teach your lesson and learn that someone had adjusted your
particular lesson the previous night.

Since we'll have all the lesson content in Git, there's no reason we
can't use pulls (or submodules, or Bower, or whatever, see #102) to
include a particular set of lesson content in the per-workshop
repository.

The way I see it, we can make maintenance easier by breaking out the
lessons and whatnot into external repositories, and then the central
workshop seed (currently swcarpentry/bc) can pull in specific versions
of those external repositories. We already do this for the Windows
installer (maintained here 1, see 2) and the installation test
scripts (maintained here 3, see [2,4]) and nobody who's using
swcarpentry/bc needs to care that “upstream” for those bits is
actually a separate repository. I'd postpone changes to the
instructor workflow until after the maintainers had settled into the
per-lesson repository workflow.q

[swaldman3]

If a group of people want to fork one of those repositories
and tweak the examples to be more relevant to economics,
they can do so;

This concerns me a little, because it sounds like we end up with duplicate versions of the lesson content (i.e. the stuff being taught) that happens to use different example data, which then risks either things getting out of sync, or extra management overhead in preventing that from happening.

Speculation on how to solve this, which is probably wrong because of my limited git knowledge: I think either the example data needs to be in a seperate repo to the other lesson content (probably over-complicated and awkward), or the various sets of example data need to be in the same repo as the lesson content. (maybe different "release" branches, where changes to lesson content get merged into all of them?).

we need volunteers who aren't experts in Git or page layout
to act as a reality check on those who are.
If you'd like to help in either capacity,
please give us a shout:
this will be one of the biggest changes we ever make,
and we'd like everyone to be part of it.

Maybe state how to give a shout? (i.e. preferred means of contact, and to whom).
Incidentally I'm not an expert in either of those things, and am happy to help so far as limited time permits. :-)

[jeixav]

Source code duplication is one obstacle I have faced when becoming acquainted with the bc repository. For example, intro.md and book/introduction.tex contain almost identical text, likewise for novice/r/00-first-timers.md and novice/r/00-first-timers.Rmd. The template in the proposal does not seem to fit the examples above. Could the proposal be revised to address examples like the ones above?

[wking]

On Mon, Sep 29, 2014 at 06:54:49AM -0700, Kristjan Onu wrote:

for novice/r/00-first-timers.md and novice/r/00-first-timers.Rmd

For the R lessons, the *.md form is built automatically from the *.Rmd
source. In my view, that means only the *.Rmd source needs to be in
the development repository, and folks can supply a built branch
(e.g. gh-pages) for folks that don't want to bother building locally.

[gvwilson]

We decided a while back to include the .md's corresponding to .ipynb's
so that non-Python people wouldn't have to install IPython to rebuild a
web site. Now that we're splitting lessons into chunks, I think it's
safer to require people to have the tools needed to build X from its raw
material. Thoughts?

[wking]

On Mon, Sep 29, 2014 at 10:50:54AM -0700, Greg Wilson wrote:

We decided a while back to include the .md's corresponding to
.ipynb's so that non-Python people wouldn't have to install IPython
to rebuild a web site. Now that we're splitting lessons into
chunks, I think it's safer to require people to have the tools
needed to build X from its raw material. Thoughts?

I don't think instructors should be responsible for building the
lessons at all. Of course, it should be easy for them to rebuild the
lessons if they want to make changes, but my preferred workflow is:

• Source (e.g. *.Rmd) in swcarpentry/some-lesson's master branch
• Compiled version ready for Jekyll (e.g. *.md) in
  swcarpentry/some-lesson's gh-pages branch
• Compiled version merging swcarpentry/some-lesson's gh-pages branch
  in swcarpentry/bc.

The development workflow would be:

Integrate a pull request:

some-lesson-maintainer$ git checkout master
some-lesson-maintainer$ git fetch origin
some-lesson-maintainer$ git merge origin/pr/123
some-lesson-maintainer$ git push origin master

When you're ready to cut a release (assuming you build things with
Make, otherwise substitute your build script):

some-lesson-maintainer$ git tag v1.2.3 master
some-lesson-maintainer$ git checkout gh-pages
some-lesson-maintainer$ git pull master
some-lesson-maintainer$ make
some-lesson-maintainer$ git add .
some-lesson-maintainer$ git commit -v 'Compile for Jekyll'
some-lesson-maintainer$ git tag v1.2.3-compiled
some-lesson-maintainer$ git push --tags origin gh-pages

Then assembly maintainers like bc's merge the compiled sources:

bc-maintainer$ git checkout gh-pages
bc-maintainer$ git pull --log git://github.com/swcarpentry/some-lesson.git v1.2.3-compiled

And instructors carry on using bc as they always have before.

[tbekolay]

I think it's safer to require people to have the tools needed to build X from its raw material.

I agree, and this would be preferred IMO. It's too easy to forget to keep two things in sync!

As for how exactly this looks in the end, I would be more comfortable with the type of workflow that @wking suggests if we could have significant portions of it automated through TravisCI. If it's as simple as someone running make and pushing that to the gh-pages branch, then it should be simple for TravisCI to do the same.

[wking]

On Mon, Sep 29, 2014 at 11:53:43AM -0700, Trevor Bekolay wrote:

If it's as simple as someone running make and pushing that to the
gh-pages branch, then it should be simple for TravisCI to do the
same.

I don't cut releases so often that it needs much automation, but if
individual maintainers feel that offloading to Travis makes sense for
their lesson, more power to 'em ;).

[gdevenyi]

Question re: repo breakup, should an instructor's guide for each topic live with the lessons in the repo?

[wking]

On Mon, Sep 29, 2014 at 02:30:56PM -0700, Gabriel A. Devenyi wrote:

Question re: repo breakup, should an instructor's guide for each
topic live with the lessons in the repo?

Yes: https://github.com/swcarpentry/site/pull/610/files#diff-ef320ce94fa57eb02be8c4a00cd63388R238

[wking]

On Sun, Sep 28, 2014 at 02:12:15PM -0700, W. Trevor King wrote:

I'm in favor of this breakup, whatever it looks like (obviously: #102
;). However, I don't really see a need to simultaneously adjust the
default procedure for consuming swcarpentry/bc.

I'm not sure on the planned timeline, but if we want to wrap up the
splitting before the new year, we'll probably have to start on the
history extraction in the next week or two. Do we want to wait on a
finalized lesson template before starting the histroy extraction (I
think not)? If any lesson maintainer teams are interested in the
pull-based integration (transparent to bc users) that I'm currently
using for the installation-test scripts [1,2,3], let me know, and we
can start in on the history extraction for your lessons.

[rgaiacs]

The second draft of our new lesson template depends on jgm/pandoc#1711.

[rbeagrie]

I have an operational question. How are we planning to handle code review for new lessons that will be their own repository? Suppose someone want to write three lessons on a new python library that they think will be useful, and they create them in a new repository. Under the old system, they could issue a pull request to get some feedback on their work. Has anyone thought about how this should work with the new repository organization?

[gvwilson]

Yup: if you're creating a brand new lesson and want some attention, you
mail 'discuss' and add it to the list on the Software Carpentry site of
"lessons you may be interested in". Think CPAN, CRAN, PyPI, etc.: you
come to SWC to find packages, you go to their sites to contribute.

[gvwilson]

Decisions have been made.