dylan-barth-stuart-coleman-a-beginners-guide-to-building-data-pipelines-with-luigi.json

{
  "description": "Growth Intelligence tracks the performance and activity of all the\ncompanies in the UK economy using their data \u2018footprint\u2019. This involves\ntracking numerous unstructured data points from multiple sources in a\nvariety of formats and transforming them into a standardised feature set\nwe can use for building predictive models for our clients.\n\nIn the past, this data was collected by in a somewhat haphazard fashion:\ncombining manual effort, ad hoc scripting and processing which was\ndifficult to maintain. In order to streamline the data flows, we\u2019re\nusing an open-source Python framework from Spotify called Luigi. Luigi\nwas created for managing task dependencies, monitoring the progress of\nthe data pipeline and providing frameworks for common batch processing\ntasks.\n",
  "duration": 2139,
  "language": "eng",
  "recorded": "2015-06-21",
  "speakers": [
    "Dylan Barth",
    "Stuart Coleman"
  ],
  "summary": "An introduction to Luigi with real life case studies showing how you\ncan break large, multi-step data processing task into a graph of\nsmaller sub-tasks that are aware of the state of their\ninterdependencies.",
  "thumbnail_url": "https://i.ytimg.com/vi/gz7tba-R7QY/hqdefault.jpg",
  "title": "A Beginner's Guide to Building Data Pipelines with Luigi",
  "videos": [
    {
      "type": "youtube",
      "url": "https://www.youtube.com/watch?v=gz7tba-R7QY"
    }
  ]
}