Readme basic usage

[KadekM]

• how to install it
• how to create simple crawler
• how to store result in file / json
• how to compose crawlers

[yelled1]

git pull the source
git clone git@github.com:KadekM/scrawler.git
** I used the sbt package to create jar files # this was the wrong move **
created a project
.
├── build.sbt (see below)
├── lib (*the compiled jar files go here ** not necessary & cause of the error **)
├── project
│   └── build.properties (specify the sbt version 0.13.15 in my case)
└── src
└── main
└── scala (*myCrawler.scala goes here)

ran intellij import by sbt
created a myCrawler.scala
copied class
and added
import com.marekkadek.scraper.Document
import com.marekkadek.scraper.jsoup.JsoupBrowser
import com.marekkadek.scrawler.crawlers.{Crawler, Visit, Yield, YieldData}
import fs2.{Strategy, Stream, Task} ** this solves two errors below **

However, I am stuck on 2 errors

1. override protected def onDocument(document: Document): Stream[Task, Yield[String]] = {
   Task Takes Type Parameters

2. Stream.emit(title) ++ Stream.emits(followableLinks)
   

Cannot resolve ++, emit, emits

Complete newbie myself. So, I am stuck here.

[yelled1]

I was able to compile it after changing
import fs2._ from import fs2.Task
But run fails!

object WikiGo {
  def main(args: Array[String]) {
    val crawler = new myCrawler
    // crawl wikipedia sequentially and take 10 elements (titles of visited websites)
    val titles: Vector[String] = crawler.sequentialCrawl("https://wikipedia.org").take(10).runLog.unsafeRun
    println(titles)
}

[IJ]> compile
[success] Total time: 0 s, completed Sep 8, 2017 11:12:51 PM
[IJ]> run
[info] Running WikiGo
[error] (run-main-5) java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
at com.marekkadek.scrawler.crawlers.Visit.(crawlers.scala:12)
at com.marekkadek.scrawler.crawlers.Crawler.sequentialCrawl(crawlers.scala:35)
at WikiGo$.main(WikiGo.scala:5)
at WikiGo.main(WikiGo.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 0 s, completed Sep 8, 2017 11:12:57 PM
[IJ]>

[visox]

Hi,

you probably have a dependancy problem, (the code is fine)

can you share with us your build.sbt ?

also you created jar file for this project ? why not just add it as a dependancy in build.sbt?

An to your first comment, this project/crawler basically emits an fs2.stream and for that you typically need to import fs2.{Strategy, Stream, Task}

[yelled1]

Hi:

Here's my build.sbt

name := "ScraperProject"

version := "1.1"

scalaVersion := "2.11.8"

libraryDependencies += "com.marekkadek" %% "scrawler" % "0.0.3"

Ah..., the jar files were somehow was causing problems!
I removed the lib directory (with jars) from the root dir & it ran fine.
I guess one cannot use libraryDependencies & jar files at the same time. My 1st Scala external lib dependent compiled & ran!
Thanks u much,

[visox]

Hi, no problem, happy crawling

[KadekM]

@yelled1 yes, just use sbt for dependency managment :)
feel free to open issue if you encounter any.