nar-kumar-chhantyal-building-data-workflows-with-luigi-and-kubernetes.json

{
  "copyright_text": "Creative Commons Attribution license (reuse allowed)",
  "description": "This talk will focus on how one can build complex data pipelines in\nPython. I will introduce Luigi and show how it solves problems while\nrunning multiple chain of batch jobs like dependency resolution,\nworkflow management, visualisation, failure handling etc.\n\nAfter that, I will present how to package Luigi pipelines as Docker\nimage for easier testing and deployment. Finally, I will go through way\nto deploy them on Kubernetes cluster, thus making it possible to scale\nBig Data pipelines on- demand and reduce infrastructure costs. I will\nalso give tips and tricks to make Luigi Scheduler play well with\nKubernetes batch execution feature.\n\nThis talk will be accompanied by demo project. It will be very\nbeneficial for audience who have some experience in running batch jobs\n(not necessarily in Python), typically people who work in Big Data\nsphere like data scientists, data engineers, BI devs and software\ndevelopers. Familiarity with Python is helpful but not needed.",
  "duration": 1764,
  "language": "eng",
  "recorded": "2019-07-11",
  "related_urls": [
    {
      "label": "Conference schedule",
      "url": "https://ep2019.europython.eu/schedule/"
    },
    {
      "label": "slides",
      "url": "https://ep2019.europython.eu/media/conference/slides/UteEqy2-building-data-workflows-with-luigi-and-kubernetes.pdf"
    }
  ],
  "speakers": [
    "Nar Kumar Chhantyal"
  ],
  "tags": [
    "Architecture",
    "Big Data",
    "Data",
    "Distributed Systems",
    "Scaling"
  ],
  "thumbnail_url": "https://i.ytimg.com/vi/41ubI6a3jzI/maxresdefault.jpg",
  "title": "Building Data Workflows with Luigi and Kubernetes",
  "videos": [
    {
      "type": "youtube",
      "url": "https://www.youtube.com/watch?v=41ubI6a3jzI"
    }
  ]
}