the-denormalized-query-engine-design-pattern.json

{
  "description": "DjangoCon US 2017 - The denormalized query engine design pattern by Simon WIllison\n\nMost web applications need to offer search functionality. Open source tools like Solr and Elasticsearch are a powerful option for building custom search engines\u2026 but it turns out they can be used for way more than just search.\n\nBy treating your search engine as a denormalization layer, you can use it to answer queries that would be too expensive to answer using your core relational database. Questions like \u201cWhat are the top twenty tags used by my users from Spain?\u201d or \u201cWhat are the most common times of day for events to start?\u201d or \u201cWhich articles contain addresses within 500 miles of Toronto?\u201d.\n\nWith the denormalized query engine design pattern, modifications to relational data are published to a denormalized schema in Elasticsearch or Solr. Data queries can then be answered using either the relational database or the search engine, depending on the nature of the specific query. The search engine returns database IDs, which are inflated from the database before being displayed to a user - ensuring that users never see stale data even if the search engine is not 100% up to date with the latest changes. This opens up all kinds of new capabilities for slicing, dicing and exploring data.\n\nIn this talk, I\u2019ll be illustrating this pattern by focusing on Elasticsearch - showing how it can be used with Django to bring new capabilities to your application. I\u2019ll discuss the challenge of keeping data synchronized between a relational database and a search engine, and show examples of features that become much easier to build once you have this denormalization layer in place.\n\nUse-cases I explore will include:\n\nFinding interesting patterns in your data\nBuilding a recommendation engine\nAdvanced geographical search and filtering\nReacting to recent user activity on your site\nAnalyzing a new large dataset using Elasticsearch and Kibana\n\nhttps://2017.djangocon.us/talks/the-denormalized-query-engine-design-pattern/",
  "language": "eng",
  "recorded": "2017-08-14",
  "speakers": [
    "Simon Willison"
  ],
  "thumbnail_url": "https://i.ytimg.com/vi/NzcvewgqYog/hqdefault.jpg",
  "title": "The denormalized query engine design pattern",
  "videos": [
    {
      "type": "youtube",
      "url": "https://www.youtube.com/watch?v=NzcvewgqYog"
    }
  ]
}