This module is part of the ISS MTech Graduate Certificate series. The module is common for both Graduate Certificate in Big Data Engineering & Web Analytics as well as Graduate Certificate in Engineering Big Data series offered by NUS-ISS. Workshops include both Spark Scala and PySpark examples.
Participants will learn various aspects of data engineering while building data intensive applications. Participants will learn to apply key practices, identify multiple data sources appraised against their business value, design the right storage, and implement proper access model(s). Finally, participants will build a scalable data pipeline solution composed of pluggable hadoop component architecture, based on the combination of requirements in a vendor/technology agnostic manner. Participants will work with Spark framework and special processing libraries.
Upon effective completion of the course, participants will be able to:
- Understand the growth of big data and need for a scalable processing framework. Understand the fundamental characteristics, storage, analysis techniques and the relevant distributions
- Understand the distributed storage essentials, storage needs, and relevant architectural mechanism in processing large amounts of structured, semi-structured and unstructured data.
- Gain expertise with the fault-tolerant computing framework (E.g. YARN) by setting up pseudo cluster nodes or cloud based nodes for processing big data. .
- Organize, store and manipulate the collected data using processing libraries. For example, using special statistical operation and stream processing data tools (E.g. Spark Special Libraries).
- Understand various data processing, querying and persistence (E.g. Spark QL APIs) available for usage in both RDD’s and Data Frame context. Perform tasks such as filtering, selection and categorization.
- Implement machine learning algorithms using Spark ML libraries.