-
Below are my solutions for the first week of learning the Spark at the Scala Academy.
-
The class SparkDemo contains our first Spark application which loads file to a DataFrame
-
The Wed class contain solution of the following exercise:
- Load CSV file
- Dataset.withColumn + functions object
- https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html
- Values should all be UPPER case
- DataFrame.show
-
The SplitWithVariableDelimiter object contains the solution of the exercise: https://jaceklaskowski.github.io/spark-workshop/exercises/sql/split-function-with-variable-delimiter-per-row.html
-
The FlatMapNumbers object contains the solution of the exercises:
-
The FlattenColumns object contains the solution of the exercise: https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Flattening-Array-Columns-From-Datasets-of-Arrays-to-Datasets-of-Array-Elements.html
-
The ConvertArrays object contains the solution of the exercise: https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Converting-Arrays-of-Strings-to-String.html
-
The DaysDiff object contains the solution of the exercise: https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Difference-in-Days-Between-Dates-As-Strings.html
-
The AddDays object contains the solution of the exercise: https://jaceklaskowski.github.io/spark-workshop/exercises/sql/How-to-add-days-as-values-of-a-column-to-date.html
-
The LimitCollect object contains the solution of the exercise: https://jaceklaskowski.github.io/spark-workshop/exercises/sql/limiting-collect_set-standard-function.html
-
The UpperColumn object contains the solution of the exercise: https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Using-upper-Standard-Function.html
-
The FindMostCommonPrefix and FindMostCommonPrefix2 objects contain two different solutions of the exercise: https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Finding-Most-Common-Non-null-Prefix-Occurences-per-Group.html
-
Notifications
You must be signed in to change notification settings - Fork 0
zdulak/spark-demo
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
About
No description, website, or topics provided.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published