Permalink
Fetching contributors…
Cannot retrieve contributors at this time
124 lines (117 sloc) 5.42 KB

Tasks

Before beta

  • ensure text parsing does not include starting newline
  • add ability to set project-specific css
  • support different fence types for examples (eg ---)
  • hello-world project generator
  • alpha for images without matching dimensions
  • record timestamp when a page started executing, use in the result template
  • Node 8
  • use pageObj.sourcePath when approving with new result instead of hard-coding the path
  • test with large images
  • test with complex page and example names
  • test with exceptions thrown throughout the test process
  • directory summaries for page folders
  • skip empty files (instead of blowing up)
  • Solve cross-linking (eg replace .md in local links to .html when generating html)
  • detect examples with the same name on a page
  • support image/* content types for content/contentType fixtures
  • Enable people to extend/plug easier
    • add more parsers
  • test overlay/positioning in other browsers
    • safari
    • firefox
  • handle gracefully cases where the expected result is provided, but the file is not present
  • report problems better when the expected result is provided, but there is no example matching that name

before 1.0

  • headless firefox
  • AWS Lambda chrome puppeteer
  • Make it easier to inspect/approve
    • CLI Approval instructions on the results page
    • CLI Approval instructions on the rendered page (for all examples)
    • Overview/Overlay/Inspect on the results page
      • set view/opacity/zoom for actual/expected/diff separately
      • scale down large images so they are easier to see on the overview
        • checkbox for original size/size to fit
  • Distinguish between a failure and error
    • record errors from fixtures
    • display error as status when listing results
    • format/test error messages
      • thrown as exceptions
      • thrown as strings
      • rejected from a promise
  • Fixture execution
    • inside node
      • detect and parse example format (eg json or yaml)
      • pass fixture parameters in the context
      • pass temporary working dir
      • allow fixtures to return a file
    • URL fixture execution (eg test the same site in various resolutions)
    • in-browser fixture execution
      • record after page load
      • record after network idle
      • record after custom event
    • webpack fixture packaging
    • AWS Lambda fixture engine
  • Extend example properties
    • Configurable clip area for screenshots
    • Configurable initial size for html windows
    • Configurable image matching precision and anti-aliasing
  • Extract configuration/enable overrides
    • Configurable source and work directories (don't just dump stuff to temp)
    • Configurable clip area for screenshots
    • Configurable initial size for html windows
    • Configurable image matching precision
    • + and anti-aliasing
    • Configurable HTML attributes (data-example)
    • Configurable HTML classes for success/failure
    • Configurable screenshot engine
    • Configurable fixture engines (via NPM modules)
    • Config file to set all options quickly
  • Server for interactive work
    • render example folder structure + README.md for folders
    • render markdown from the examples folder
    • offer running the spec if the MD contains any examples
    • approving changes
    • Running tests by directly reading specs from github (eg public site, connect to a gitub repo)
    • Approving by directly committing to github? (eg for specs stored in github repos)
  • Result formatters
    • JSON
    • junit
    • tap
  • Reporters
    • ASCII console
    • ANSI color console
    • Summary (dot)
  • CLI/NPM tasks
    • executing tests
    • run all tests
    • run a specific page
    • run a specific example
    • exit with -1 in case of errors/failures
    • approving changes
    • approve a whole page
    • approve a specific example
    • approve all failed examples

Released in Alpha

  • breadcrumbs on rendered page -> links
  • Start/stop chrome once per run, not once per page
  • navigation on results page (breadcrumbs and page links)
  • Retina screen sizing
  • change sync FS operations to async
  • put example name into the example object/results object for easier templating
  • use timestamps for executed and started in result template
  • test with multiple examples in a single file
  • test with multiple files
  • test with files in subfolders
  • Choose a markdown rendering system (https://github.com/markdown-it)
  • Choose a Node DOM manipulation system (https://github.com/cheeriojs/cheerio)
  • Build local screenshots (using Chrome Headless and https://github.com/cyrus-and/chrome-remote-interface)
  • Choose an image comparison system (https://github.com/mapbox/pixelmatch)
  • generate a combined image if dimensions do not match, showing both images with alpha
  • Add a generic layout to HTML when converting from MD
  • Add a generic template for result files
  • Create an index with all test results
  • Clean up templates and provide a nice layout
  • documentation on key usage scenarios
  • Run examples without expected outcomes (fail + show actual)