Skip to content

Latest commit

 

History

History
73 lines (49 loc) · 3.1 KB

going-big-with-pandas_alexandre-carvalho.md

File metadata and controls

73 lines (49 loc) · 3.1 KB
Raw

Going BIG with Pandas

  • Speaker : Alexandre Carvalho
  • Length : 2h with an interval
  • Language : English or Portuguese

This is a workshop

Description

This is a Workshop to explore BIG Data using python pandas and other tools (or how to efficiently deal with lots of data in a laptop) The keyword of the workshop will be: efficiency

pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. It's being widely used by scientists across multiple fields.

If you want a hands-on experience on how to use pandas, and how it can befit you, this is the workshop to go! I've used it to cleanup data, transform it, and create each user anonymized profile. Later it was also easy to process profile based recommendations, where I would score 10k items on a laptop in under 60ms.

We're going to explore the basic API's of pandas to do in memory transformation, filter, groups, joins, etc... In the process we will learn about jupyter notebooks which are also being used in science not only to do trial and errors, but also to share the results and how they were achieved, enabling repeatability.

For the workshop we will process generated data for a possible EPG and Activity Log (each attendant will generate their own data, so no repeatability here :D ). We will then extract data from the log, and process it. In the end of the workshop we should be able to give item to item recommendations based on what people watched...

Main tools that will be used:

  • python 3 (3.6 if you can...)
  • jupyter notebooks
  • numpy
  • pandas

Please take some time to setup your environment prior to this workshop,... take a look at ajscarvalho/pandas_workshop on github Also, please ensure that you have hard disk space, because we could end up generating some big files.

Speaker Bio

Alexandre Carvalho

Speaker Image

While Working

  • Defined and built a CRM system from zero in PHP, javascript an vbscript, back in

  • Created a configurable automatic task executer with a limited set of actions (like a software robot), similar to a very limited code interpreter. (PHP, Oracle). Also created an Asynchronous Task Execution Environment.

  • Built Mobile, Responsive and IPTV sites while on SAPO

  • Currently: System Architect at NOS defining IPTV apps, Transversal systems and diving deep in bigdata, A.I., etc.

On my Free Time

  • Canvas and Node.Js exploration resulting in a multiplayer game [2010] (buggy webworkers version, with live code editing here: http://carvalho.sl.pt/space_djs/djs.html - just follow Test Tutorial and start at mission 1)
  • Replicating mobile apps as websites (responsive)
  • Arduino and rPi experiments
  • Domotics (Home blinds control web app)
  • Beacon experiments
  • Aikido

Links

Click here to see the full calendar and pick your favorite talks