jiaqi-liu-building-a-data-pipeline-with-testing-in-mind-pycon-2018.json

{
  "copyright_text": "Standard YouTube License",
  "description": "It\u2019s one thing to build a robust data pipeline process in python but a whole other challenge to find tooling and build out the framework that allows for testing a data process. In order to truly iterate and develop a codebase, one has to be able to confidently test during the development process and monitor the production system.\n\nIn this talk, I hope to address the key components for building out end to end testing for data pipelines by borrowing concepts from how we test python web services. Just like how we want to check for healthy status codes from our API responses, we want to be able to check that a pipeline is working as expected given the correct inputs. We\u2019ll talk about key features that allows a data pipeline to be easily testable and how to identify timeseries metrics that can be used to monitor the health of a data pipeline.",
  "duration": 1215,
  "language": "eng",
  "recorded": "2018-05-11",
  "related_urls": [
    {
      "label": "Conference schedule",
      "url": "https://us.pycon.org/2018/schedule/talks/"
    },
    {
      "label": "Conference slides (SpeakerDeck)",
      "url": "https://speakerdeck.com/pycon2018"
    },
    {
      "label": "talk schedule",
      "url": "https://us.pycon.org/2018/schedule/presentation/161/"
    }
  ],
  "speakers": [
    "Jiaqi Liu"
  ],
  "tags": [],
  "thumbnail_url": "https://i.ytimg.com/vi/fRUCAt9WFpc/maxresdefault.jpg",
  "title": "Building a Data Pipeline with Testing in Mind",
  "videos": [
    {
      "type": "youtube",
      "url": "https://www.youtube.com/watch?v=fRUCAt9WFpc"
    }
  ]
}