Spark Streaming's Kafka libraries not found in class path. Try one of the following.

[Lamonkey]

Spark Streaming's Kafka libraries not found in class path. Try one of the following.

1. Include the Kafka library and its dependencies with in the
   spark-submit command as

   $ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8:2.4.4 ...

2. Download the JAR of the artifact from Maven Central http://search.maven.org/,
   Group Id = org.apache.spark, Artifact Id = spark-streaming-kafka-0-8-assembly, Version = 2.4.4.
   Then, include the jar in the spark-submit command as

   $ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...

[Tony-Hsieh]

I currently set Spark version from 3.0.0 to 2.4.4 on jupytor notebook
!pip install pyspark==2.4.4
and run the code below
!spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.4.4 test.py
test.py is the file that run spark with kafka.
It still has some problems inside but you can try if it works out.