Scala And Spark

Welcome ! Because bookemarks are too confidentials, at Paris 13 university, we decided to share every usefull links we fall on during our ongoing learning of Scala and Spark, trying to organize them as well as we can. Don't hesitate to add cool links we have missed.

Some comments will be added with time 😄 !

Scala

Use it wisely, don't forget to go for a tour and if you're ready go directly to the famous awesome-scala github regrouping most starred scala projects.

Learn by practicing

• Learn Scala standard library and some frameworks through exercising

Overviews

• Because we have to begin somewhere
• The Neophyte's Guide to Scala
• Tweeter Scala School
• Scala cheatsheet from a java point of view

Specifics aspects

• Keywords

  • for with conditions
    • alvinalexander Blog
  • for yield
    • alvinalexander Blog
  • lazy
    • What does a lazy val do ?
    • Interesting explanations
  • implicits
    • Scala tour
    • Where does Scala look for implicits ?
    • implicit design pattern

• Operator

• Scala @ operator

• Collection

  • Some practical advices
  • Hints about collection operations complexity
  • collect or how to filter and map in one pass

• Class

  • Overview

• Option or the nice way to avoid null

  • http://danielwestheide.com/blog/2012/12/19/the-neophytes-guide-to-scala-part-5-the-option-type.html
  • http://code.scottshipp.com/2015/04/17/scala-using-option-can-still-result-in-a-npe/

• Closure

  • Here is an example and an explanation.

• Generics

  • Class
  • Functions
  • Type Constructor Polymorphism
  • Generics of a Higher Kind

• Tail recursiv functions

• Covariance and Contravariance

  • Scala tour**
  • http://blog.kamkor.me/Covariance-And-Contravariance-In-Scala/
  • https://twitter.github.io/scala_school/type-basics.html
  • https://apiumhub.com/tech-blog-barcelona/scala-generics-covariance-contravariance/
  • https://www.atlassian.com/blog/software-teams/covariance-and-contravariance-in-scala
  • http://julien.richard-foy.fr/blog/2013/02/21/be-friend-with-covariance-and-contravariance/

• Concurency

  • https://blog.mathewrathbone.com/2017/03/28/scala-concurrency-options.html

• Did you said monad ?

  • http://blog.brakmic.com/writing-monads-in-scala-with-spark-notebook/

• Annotations

  • @transient
  • @specialized
    • Type specialization
    • One StackOverFlow
    • Another blog post on it
    • More advanced examples

Scala sacred writtings

• Generics of a Higher Kind
• Fighting Bit Rot with Types (Experience Report: Scala Collections)
• Type Classes as Objects and Implicits

Frameworks

• Awesome scala, refer many many awesome projects in scala
• Smile 😄 which is way more than a nicely named Scala/Java library for machine learning.
• Scalaz, an extension of the standard scala library that allow to do pure functional programming
  • Multiple ressources
  • An introduction
  • An adventure to learn scalaz
• Breeze
  • API
  • Quickstart
  • Linear Algebra Cheat Sheet
• Shapeless
  • Introduction to shapeless
• Spire
• Scala.js
• Monocle

Benchmarking Scala collections

Because we hardly desire to know the commodities comming with our favorite scala's collections here are some linked ressources:

• Benchmarking Scala Collections
• Performance Comparison between immutable Seq, List, Vector

Design Patterns

• Awesome article by Hayoi
• Another cool article on multiple patterns
• The @transient lazy pattern
• A master thesis on different patterns
• Implicits patterns
• Patterns with implementation
• Stackable Trait Pattern
• Type Classes
  • Example 1
  • Example 2
  • Example 3
• Scala Design Patterns 2nd Edition
  • A must have !

Miscellaneous

• flatMap to Option
• Union Types
• Equivalence between Java and Scala
• Returning the "Current" Type
• Advanced Types
• 99 Scala Problems
• One of many advantage of Spire
• Higher-kinded types: the difference between giving up, and moving forward
• A simpler way to returning the "current" type in Scala
• Curious case of putting override modifier when overriding an abstract method in Scala

Sbt

• Multi-projects
  • An example of a sbt multi-project
  • A bit of explanations
• sbt publish errors while compile works, try to clean you .ivy2/local/yourProject

Before introduce Spark, i will recomand to you the SparkNotebook

So why introduce a first appearance Spark tool, simply because it offers multiple usefull functionalities to practice as well Scala as Spark for any kind of purposes. You will find complementary informations on the SparkNotebook creator compagny blog.

Spark

Let's start with the official website guides without forgetting to give the link towards the Databricks blog and for impatients the usefull awesome-spark and spark-packages links.

General learning purpose

• https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
• https://www.gitbook.com/book/jaceklaskowski/mastering-apache-spark/details

Specifics aspects

• Caching and Persistance
• Broadcast variables
• Accumulators
• ByKey methods
  • ReduceByKey
    • How it works
  • Why avoid GroupByKey
  • AggregateByKey
  • CombineByKey
    • Examples one two and three
• Why prefer mapPartitions over map

MLLib examples

• Pipelines
• Some troubleshootings
  • Use non serializable objects
  • Heavy workloads common issues

Miscellaneous

• Parse your csv
• Adjust your log levels
• Tuning your spark applications

Cool projects

• TSNE on spark, the TSNE is a dimentional reduction technique from the manifold family
• LIPN on Spark
  • This is what we are making to Paris 13 University on Spark

Garbage collector optimizations

General informations and tuning

• Managing Memory and Garbage Collection
• Minimize Java Memory Usage with the Right Garbage Collector
• Tuning G1GC For SOA
• G1Gc Tuning
• Others G1 GC tests

Spark applications

• Tuning Java Garbage Collection for Apache Spark Applications

Awesome scala links and references

Here are listed some blogs from where cames links in this page, thanks to these guys we can learn more easily so thank you for you hard work dear scala fellas 😉.

• Books
  • Programming in Scala 3rd Edition let's start by Scala's father last edition book.
  • Functional Programming in Scala
  • A Spark online book
  • Scala Design Patterns 2nd Edition
• Blogs
  • Twitter Scala School
  • An awesome blog
  • Scala Cookbook author blog
  • http://danielwestheide.com/blog/archives/
  • Databricks blog

Scala.js

• Hands on Scala.js
• Scala-js-d3
• Getting started

Scala 3

• Medium blog post