Skip to content

Initial notes

Jump to bottom
Andrew Stanton edited this page Nov 19, 2018 · 78 revisions

Types of transformations

Substitution

  • RegEx
  • String

Conversions between formats

  • HTML <==> MD

HTML tidying

  • Remove attributes (all | some (whitelist | blacklist))
  • Styles
  • Detect attributes duplicate
  • Detect and fix syntax errors E.g. li not in a ul
  • Empty elements

Links

  • Check validity of the link
  • Remove / Add attributes
  • Remove / Add URL paths | args | anchors
  • Encode characters
  • Take data and make it a valid link. Need to support
    • URL
    • Email
    • Phones

Other

  • Sorting & grouping
    • Alphabetical
    • Numeric
    • Custom orders
    • Nested sorting
  • Formatting
    • Text
      • Case
    • Code
  • Email | Phone | URL detection and processing
  • Check links
  • URL processing
  • Break on delimiter
    • ,
    • ;
      • (all the dashes?)

Analysis

  • Validating
    • Text (Spelling, grammar, other metrics)
    • Code (Linting)
  • Compare text
  • Search for terms

Data IO

  • APIs
  • Browser plugin
  • API calls
    • Built in
    • User created
  • Import / export to/from
    • services
    • Data formats

Web scraping

  • Extract content
  • Validate pages
    • Checks
      • Presence
      • Values
      • Order of components
    • Data source
      • From given list
      • Extract from search results

Advanced

  • Process delimited blocks (identify components)
  • JSON
    • JSONata?

Other

  • Logic
  • APIs
  • Grouping transformations into named groups
  • Allow for input / output values to be left blank and asked for at run time
  • Scheduled events
  • Context data
  • Branching paths Dependant on conditions
  • Pagination / breaking down large collections
  • Workflows. Load data work through each of them in turn. Could include manual tweak?

Sentinel feature

  • Check matching webpages for absence of / undesired content
  • Could actively ask or log it as a potential issue
  • Integrate with existing services

Logging?

Flag all issues detected. Even if they aren't to be processed. Useful for teaching people about the errors.

Diffing

  • Text values
  • Mark up
    • Attributes
    • Styling

Acquisition Data from different sources

  • Merge & processing
  • Data to be displayed when other sources are open in the browser

Interactive units

Wrapping elements

Load editor tools into text areas on websites

  • Call workflows
  • Editing tools
  • Use tab to insert the character

Dashboard

  • Analytics summaries
  • List of scraped websites
  • Team resources

Time tracking? How long to get something done

Compare text

  • Identical (Case sensitivity?)
  • Similarity
  • Diff
  • Order of things

Smarter search on webpages

  • Fallback to fuzzy if it doesn't have a perfect match
    • E.g. Not fail due to SCs fear of commas
  • Compare against data sources
  • Extract menus

Shared clipboard

  • Constant stuff
  • Named attributes from extracted data

Detect duplicate elements

Prompt for useful follow up components once one has been added

Vim mode

Autocomplete for text areas Smarter versions of Ctrl Arrow keys Join options

Extract salient info - useful values

Strip out Google paste bloat

Text case transformation Blacklist words

Build up a collection of named pieces of data

Autocomplete Fed from data source or added each time new data is used

Document / automate internal policy E.g. Internal links same page External in new tab + useful title text

Analysis Data visualisations Display links between pages / data

Bundles with domain specific values

Give URL get title from page and use as a text

Help with context switching

  • Save notes
  • Bookmark exact location in resources
  • Handover tools

Semantic highlighting Object Schemas

Take over editor elements. Templates + look up & forms Drag and drop reordering

Variant extraction

  • Different courses
  • The names represented in different formats

Clone this wiki locally