Here you can find the materials for the "Data Engineering 3: Batch Jobs and APIs" course, part of the MSc in Business Analytics at CEU. For the previous editions, see 2017/2018, 2018/2019, 2019/2020, 2020/2021, 2021/2022, 2022/2023.

Table of Contents

• Table of Contents
• Schedule
• Location
• Syllabus
• Technical Prerequisites
• Class Schedule
• Homeworks
• Home assignment
• Getting help

Schedule

2 x 3 x 100 mins on March 20 and 27:

• 13:30 - 15:10 session 1
• 15:10 - 15:40 break
• 15:40 - 17:20 session 2
• 17:20 - 17:40 break
• 17:40 - 19:20 session 3

Location

In-person at the Vienna campus (QS B-421).

Syllabus

Please find in the syllabus folder of this repository.

Technical Prerequisites

1. You need a laptop with any operating system and stable Internet connection.
2. Please make sure that Internet/network firewall rules are not limiting your access to unusual ports (e.g. 22, 8787, 8080, 8000), as we will heavily use these in the class (can be a problem on a company network). CEU WiFi should have the related firewall rules applied for the class.
3. Please install Slack, and join the #ba-de3-2023 channel in the ceu-bizanalytics group.
4. When joining remotely, it's highly suggested to get a second monitor where you can follow the online stream, and keep your main monitor for your own work. The second monitor could be an external screen attached to your laptop, e.g. a TV, monitor, projector, but if you don't have access to one, you may also use a tablet or phone to dial-in to the Zoom call.

Class Schedule

Will be updated from week to week.

Week 1

Goal: learn how to run and schedule R jobs in the cloud.

Background: Example use-cases and why to use R in the cloud?

Excerpts from https://daroczig.github.io/talks

• "A Decade of Using R in Production" (Real Data Science USA - R meetup)
• "Getting Things Logged" (RStudio::conf 2020)
• "Analytics databases in a startup environment: beyond MySQL and Spark" (Budapest Data Forum 2018)

Welcome to AWS!

1. Use the following sign-in URL to access the class AWS account: https://657609838022.signin.aws.amazon.com/console

2. Secure your access key(s), other credentials and any login information ...

   ... because a truly wise person learns from the mistakes of others!

   "When I woke up the next morning, I had four emails and a missed phone call from Amazon AWS - something about 140 servers running on my AWS account, mining Bitcoin" -- Hoffman said

   "Nevertheless, now I know that Bitcoin can be mined with SQL, which is priceless ;-)" -- Uri Shaked

   So set up 2FA (go to IAM / Users / username / Security credentials / Assigned MFA device): https://console.aws.amazon.com/iam

   PS probably you do not really need to store any access keys, but you may rely on roles (and the Key Management Service, and the Secrets Manager and so on)

3. Let's use the eu-west-1 Ireland region

Getting help

Contact Gergely Daroczi on the ceu-bizanalytics Slack channel or open a GitHub ticket.