Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

first cut at createRemakefile #281

Merged
merged 2 commits into from
Oct 16, 2017
Merged

Conversation

aappling-usgs
Copy link
Member

@aappling-usgs aappling-usgs commented Oct 14, 2017

First cut at createRemakefile() function.

  • After running createRemakefile() in USGS-VIZLAB/example, it's [mostly] possible to run remake::make(remake_file='remake.yaml') and build the entire example project! That makes for a good start on Switch to remake? #246.
  • The [mostly] caveat from the previous bullet is that when I run remake::make(remake_file='remake.yaml') in a fresh session, things break unless I manually source those files listed in sources:. Why would this be? I don't understand.
  • Creates a single remake.yaml for all of the blocks, and looks for cross references in any phase (not just the current one), so also addresses Allow fetch to depend on process item #88 (but testing would be nice)
  • Includes recipes for those resources that are depended on by other targets, so also addresses tell make the provenance of resources from vizlab #215 (but are these resources now getting copied over twice, once by remake and once by vizlab? more inspection & testing needed here, too)
  • Does not yet reproduce the work done for rebuild necessary items based on changes to viz.yaml non-file options #193 (depending on the viz.yaml blocks). This can be slick once we implement it, because the dependencies can be R objects instead of files, but there's nothing there yet.
  • Does not yet implement timestamp handling for fetch - just calls fetch on every item.
  • There's a job group (group target) for the overall viz, but I'd still like to add job groups for fetch, process, etc. individually

Here's the remake.yaml that gets built for USGS-VIZLAB/example using this PR:

# Do not edit - automatically generated
# from the vizlab remake.mustache template

target_default: viz

packages:
  - vizlab
  - dplyr
  - dataRetrieval

sources:
  - scripts/fetch/cars.R
  - scripts/process/cars.R
  - scripts/process/cuyahoga.R
  - scripts/visualize/visualize.R


targets:

  # --- parameter --- #
  
  plot-info:
    command: parameter(I('plot-info'))

  color-stuff:
    command: parameter(I('color-stuff'))

  # --- fetch --- #
  
  'data/iris.csv':
    command: fetch(I('iris-data'))

  'cache/cars.csv':
    command: fetch(I('cars-data'))

  'cache/fetch/CuyahogaTDS.csv':
    command: fetch(I('Cuyahoga'))

  'data/markdownText.yaml':
    command: fetch(I('markdown-text'))

  # --- process --- #
  
  'cache/car-loess.rds':
    command: process(I('calc-cars'))
    depends:
      - 'cache/cars.csv'

  'cache/process/CuyahogaShort.tsv':
    command: process(I('CuyahogaShort'))
    depends:
      - 'cache/fetch/CuyahogaTDS.csv'

  # --- visualize --- #
  
  'figures/cars.png':
    command: visualize(I('plot-cars'))
    depends:
      - 'cache/car-loess.rds'
      - 'cache/cars.csv'
      - plot-info

  'figures/iris.png':
    command: visualize(I('plot-iris'))
    depends:
      - 'data/iris.csv'

  'figures/cuyahogaFig.svg':
    command: visualize(I('cuyahogaFig'))
    depends:
      - 'cache/process/CuyahogaShort.tsv'
      - color-stuff

  # --- publish --- #
  
  index:
    command: publish(I('index'))
    depends:
      - sourcesansprofont
      - cars-section
      - social-section
      - vertical-social-section
      - footer-section
      - header-section
      - 'layout/css/main.css'
      - text-section
      - iris-section
      - Cuyahoga-section
      - markdown-section1
      - markdown-section2
      - 'target/css/footer.css'
      - 'target/css/header.css'
      - 'target/css/socialMedia.css'
      - 'target/js/jquery-3.2.1.min.js'
      - 'target/js/svg-injector-1.1.3.min.js'
      - 'target/js/vizlab.js'

  cars-section:
    command: publish(I('cars-section'))
    depends:
      - 'figures/cars.png'

  text-section:
    command: publish(I('text-section'))

  markdown-section1:
    command: publish(I('markdown-section1'))
    depends:
      - 'data/markdownText.yaml'

  markdown-section2:
    command: publish(I('markdown-section2'))
    depends:
      - 'data/markdownText.yaml'

  iris-section:
    command: publish(I('iris-section'))
    depends:
      - 'figures/iris.png'

  Cuyahoga-section:
    command: publish(I('Cuyahoga-section'))
    depends:
      - 'figures/cuyahogaFig.svg'

  'layout/css/main.css':
    command: publish(I('figure-style'))

  social-section:
    command: publish(I('social-section'))
    depends:
      - 'target/css/socialMedia.css'

  vertical-social-section:
    command: publish(I('vertical-social-section'))
    depends:
      - 'target/css/socialMedia.css'

  header-section:
    command: publish(I('header-section'))
    depends:
      - 'target/images/usgsLogo.png'

  footer-section:
    command: publish(I('footer-section'))
    depends:
      - 'target/css/footer.css'

  sourcesansprofont:
    command: publish(I('sourcesansprofont'))

  # --- resource --- #
  
  'target/images/usgsLogo.png':
    command: publish(I('lib-usgs-logo'))

  'target/js/vizlab.js':
    command: publish(I('lib-vizlab-javascript'))

  'target/css/header.css':
    command: publish(I('lib-header-css'))

  'target/css/footer.css':
    command: publish(I('lib-footer-css'))

  'target/css/socialMedia.css':
    command: publish(I('lib-social-media-css'))

  'target/js/svg-injector-1.1.3.min.js':
    command: publish(I('lib-svg-injector-js'))

  'target/js/jquery-3.2.1.min.js':
    command: publish(I('lib-jquery-js'))



  # --- Job groups --- #

  viz:
    depends: 
      - plot-info
      - color-stuff
      - 'data/iris.csv'
      - 'cache/cars.csv'
      - 'cache/fetch/CuyahogaTDS.csv'
      - 'data/markdownText.yaml'
      - 'cache/car-loess.rds'
      - 'cache/process/CuyahogaShort.tsv'
      - 'figures/cars.png'
      - 'figures/iris.png'
      - 'figures/cuyahogaFig.svg'
      - index
      - cars-section
      - text-section
      - markdown-section1
      - markdown-section2
      - iris-section
      - Cuyahoga-section
      - 'layout/css/main.css'
      - social-section
      - vertical-social-section
      - header-section
      - footer-section
      - sourcesansprofont
      - 'target/images/usgsLogo.png'
      - 'target/js/vizlab.js'
      - 'target/css/header.css'
      - 'target/css/footer.css'
      - 'target/css/socialMedia.css'
      - 'target/js/svg-injector-1.1.3.min.js'
      - 'target/js/jquery-3.2.1.min.js'

@coveralls
Copy link

Coverage Status

Coverage decreased (-4.07%) to 69.579% when pulling de854b6 on aappling-usgs:remake into 2afe9f1 on USGS-VIZLAB:remake.

@aappling-usgs
Copy link
Member Author

@ldecicco-USGS - I merged this by accident - your PR review will still be welcome.

I've made notes about the consequences of this PR for #88 and #215, added a list of remaining to-do items at the bottom of #246, and spun off two new issues (#287 and #288) for tasks I think we can treat as distinct from #246 (and that could be done by you or @limnoliver if interested).

Because this PR leaves all those issues open, I've put it in a new remake branch on USGS-VIZLAB/vizlab where we can continue to collaborate on it until it's ready to merge into master.

@ldecicco-USGS
Copy link
Contributor

Lookin' good! I don't know if it helps or not, but I don't think paths need quotes (ie 'figures/cuyahogaFig.svg' could be figures/cuyahogaFig.svg

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants