`spider_skips` property broken #401

paullew · 2016-03-14T02:11:11Z

I'm using the default spider.yaml from https://raw.githubusercontent.com/BBC-News/wraith/master/templates/configs/spider.yaml

I've tried running it with wraith installed locally on my mac, and also via the wraith docker image. Both fail, with different error messages.

On my mac locally:

$ wraith capture spider.yaml
Config validated. No serious issues found.
no paths defined in config, crawling from site root
creating new spider file
/Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:298:in `=~': type mismatch: String given (TypeError)
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:298:in `block in skip_link?'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:298:in `any?'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:298:in `skip_link?'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:256:in `visit_link?'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:151:in `block in run'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:151:in `delete_if'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:151:in `run'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:92:in `block in crawl'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:83:in `initialize'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:90:in `new'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:90:in `crawl'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/anemone-0.7.2/lib/anemone/core.rb:18:in `crawl'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/spider.rb:69:in `spider'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/spider.rb:35:in `determine_paths'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/spider.rb:23:in `check_for_paths'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/cli.rb:36:in `check_for_paths'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/cli.rb:133:in `block in capture'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/cli.rb:28:in `within_acceptable_limits'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/lib/wraith/cli.rb:130:in `capture'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/gems/wraith-3.1.0/bin/wraith:5:in `<top (required)>'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/bin/wraith:23:in `load'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/bin/wraith:23:in `<main>'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `eval'
    from /Users/paullew/.rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `<main>'

Running it via the wraith docker image:

$ docker run --rm -P -v ~/devel/resources/testing/wraith:/wraithy -w='/wraithy' bbcnews/wraith capture spider.yaml
/usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/spider.rb:64:in `spider': undefined local variable or method `wraith' for #<Wraith::Crawler:0x005648d0ca4be8> (NameError)
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/spider.rb:36:in `determine_paths'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/spider.rb:24:in `check_for_paths'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/cli.rb:36:in `check_for_paths'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/cli.rb:134:in `block in capture'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/cli.rb:28:in `within_acceptable_limits'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/cli.rb:131:in `capture'
    from /usr/local/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /usr/local/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /usr/local/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /usr/local/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/bin/wraith:5:in `<top (required)>'
    from /usr/local/bin/wraith:23:in `load'
    from /usr/local/bin/wraith:23:in `<main>'
Config validated. No serious issues found.
no paths defined in config, crawling from site root

The text was updated successfully, but these errors were encountered:

altV · 2016-03-18T12:00:24Z

I'm getting undefined local variable or method `wraith' for #Wraith::Crawler:0x005648d0ca4be8 (NameError) for default spider.yml as well, version 3.1.2

imagreenplant · 2016-03-27T20:49:51Z

Same here:

wraith capture configs/spider.yaml Config validated. No serious issues found. no paths defined in config, crawling from site root /Library/Ruby/Gems/2.0.0/gems/wraith-3.1.2/lib/wraith/spider.rb:64:inspider': undefined local variable or method wraith' for #<Wraith::Crawler:0x007fd2f1938610> (NameError)

Vexrm · 2016-03-28T14:56:39Z

Ditto. I can't add extra information, but am watching for more info.

trioni · 2016-03-31T08:14:39Z

I get the same thing. Same error as @imagreenplant

Dbuggerx · 2016-06-06T11:51:56Z

I'm also getting the same error as @imagreenplant. Is this solved?

ocrunch · 2016-06-08T21:38:06Z

Same Issue!

slimatic · 2016-06-09T03:49:45Z

Any thoughts on what this issue could be?

/usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.2.0/lib/wraith/spider.rb:64:inspider': undefined local variable or method wraith' for #<Wraith::Crawler:0x0055cde740ed58> (NameError)

xinbin · 2016-06-14T18:27:29Z

I have the same error message as @slimatic , full dump:

/usr/local/lib/ruby/gems/2.2.0/gems/wraith-3.1.2/lib/wraith/spider.rb:64:in
 'spider': undefined local variable or method 'wraith' for # (NameError)
    from /usr/local/lib/ruby/gems/2.2.0/gems/wraith-3.1.2/lib/wraith/spider.rb:36:in 'determine_paths'
    from /usr/local/lib/ruby/gems/2.2.0/gems/wraith-3.1.2/lib/wraith/spider.rb:24:in 'check_for_paths'
    from /usr/local/lib/ruby/gems/2.2.0/gems/wraith-3.1.2/lib/wraith/cli.rb:36:in 'check_for_paths'
    from /usr/local/lib/ruby/gems/2.2.0/gems/wraith-3.1.2/lib/wraith/cli.rb:134:in 'block in capture'
    from /usr/local/lib/ruby/gems/2.2.0/gems/wraith-3.1.2/lib/wraith/cli.rb:28:in 'within_acceptable_limits'
    from /usr/local/lib/ruby/gems/2.2.0/gems/wraith-3.1.2/lib/wraith/cli.rb:131:in 'capture'
    from /usr/local/lib/ruby/gems/2.2.0/gems/thor-0.19.1/lib/thor/command.rb:27:in 'run'
    from /usr/local/lib/ruby/gems/2.2.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in 'invoke_command'
    from /usr/local/lib/ruby/gems/2.2.0/gems/thor-0.19.1/lib/thor.rb:359:in 'dispatch'
    from /usr/local/lib/ruby/gems/2.2.0/gems/thor-0.19.1/lib/thor/base.rb:440:in 'start'
    from /usr/local/lib/ruby/gems/2.2.0/gems/wraith-3.1.2/bin/wraith:5:in ''
    from /usr/local/bin/wraith:23:in 'load'
    from /usr/local/bin/wraith:23:in ''

bjorndavis · 2016-07-06T18:20:54Z

I'm getting this as well:

C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/wraith-3.2.0/lib/wraith/spider.rb:65:in `spider': undefined local variable or met
hod `wraith' for #<Wraith::Crawler:0x00000002e81ca8> (NameError)
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/wraith-3.2.0/lib/wraith/spider.rb:38:in `determine_paths'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/wraith-3.2.0/lib/wraith/spider.rb:24:in `check_for_paths'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/wraith-3.2.0/lib/wraith/cli.rb:36:in `check_for_paths'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/wraith-3.2.0/lib/wraith/cli.rb:134:in `block in capture'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/wraith-3.2.0/lib/wraith/cli.rb:28:in `within_acceptable_limits'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/wraith-3.2.0/lib/wraith/cli.rb:131:in `capture'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/wraith-3.2.0/bin/wraith:5:in `<top (required)>'
        from C:/Ruby22-x64/bin/wraith:23:in `load'
        from C:/Ruby22-x64/bin/wraith:23:in `<main>'

catchergeese · 2016-07-11T07:03:38Z

It fails for me too (default spider config yaml file, running in docker container):

$ wraith capture configs/spider.yaml
/usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.2.0/lib/wraith/spider.rb:64:in `spider': undefined local variable or method `wraith' for #<Wraith::Crawler:0x0055808ac4a508> (NameError)
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.2.0/lib/wraith/spider.rb:36:in `determine_paths'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.2.0/lib/wraith/spider.rb:24:in `check_for_paths'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.2.0/lib/wraith/cli.rb:36:in `check_for_paths'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.2.0/lib/wraith/cli.rb:134:in `block in capture'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.2.0/lib/wraith/cli.rb:28:in `within_acceptable_limits'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.2.0/lib/wraith/cli.rb:131:in `capture'
    from /usr/local/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /usr/local/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /usr/local/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /usr/local/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.2.0/bin/wraith:5:in `<top (required)>'
    from /usr/local/bin/wraith:23:in `load'
    from /usr/local/bin/wraith:23:in `<main>'

Are there any chances you are going to fix it in the near future?

kyleskrinak · 2016-08-29T19:43:08Z

3.2.1 does not address this issue on my system. I'm still seeing the error message:

.rvm/gems/ruby-2.2.0/gems/anemone-0.7.2/lib/anemone/core.rb:298:in `=~': type mismatch: String given (TypeError)
        from /Users/x/.rvm/gems/ruby-2.2.0/gems/anemone-0.7.2/lib/anemone/core.rb:298:in `block in skip_link?'
        from /Users/x/.rvm/gems/ruby-2.2.0/gems/anemone-0.7.2/lib/anemone/core.rb:298:in `any?'
        etc…

ChrisBAshton · 2016-08-29T19:51:48Z

This error has been fixed in 3.2.1:

'spider': undefined local variable or method 'wraith' for # (NameError)

However, I can see that the original error in this issue is:

core.rb:298:in =~': type mismatch: String given (TypeError)

This issue has been closed in error. Re-opening.

Peter-Petrik · 2016-10-11T04:08:09Z

Experiencing this in 3.2.1

/var/lib/gems/1.9.1/gems/anemone-0.7.2/lib/anemone/core.rb:298:in```=~': type mismatch: String given (TypeError)

In spider.yaml I commented out:
- !ruby/regexp /^\/baz\//
and I'm no longer seeing the error.

sembrat · 2018-05-08T20:17:59Z

For the folks encountering this issue, does the removal of non-regexp within the spider_skips fix this?

Some quick testing on my end noticed that anything processed within Ruby regexp was handled fine, whereas anything with a string of the path broke spider_skips.

To solve this, I had to encapsulate all my non-regexp strings to skip as strict string matches in regexp, which isn't exactly ideal for those (like me) who are awful at regexp syntax.

edurenye · 2018-11-30T09:15:26Z

This error is related to the web-spider framework that is using wraith called Anemone this framework in not maintained anymore, last commit was in 2012.
I think we should replace it with Medusa a maintained fork of Anemone that has the same API, so should work without to much trouble.

ErroneousBosch · 2019-03-13T17:50:29Z

@sembrat

Yes, switching to regex does seem to correct the errors.

ChrisBAshton added the Spider mode label Apr 8, 2016

ChrisBAshton mentioned this issue Apr 15, 2016

Spider mode not working with wraith #422

Closed

Domi202 mentioned this issue Jul 13, 2016

Fix "undefined local variable or method `wraith'" #447

Merged

ChrisBAshton closed this as completed in #447 Aug 4, 2016

ChrisBAshton reopened this Aug 29, 2016

ChrisBAshton changed the title ~~Default spider.yaml fails with error~~ spider_skips property broken Nov 25, 2016

mramitanand mentioned this issue Jan 10, 2017

unable to spider - fails on imported config check #498

Closed

4 tasks

edurenye mentioned this issue Nov 30, 2018

Replace Anemone for Medusa web-spider framework #605

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`spider_skips` property broken #401

`spider_skips` property broken #401

paullew commented Mar 14, 2016

altV commented Mar 18, 2016

imagreenplant commented Mar 27, 2016

Vexrm commented Mar 28, 2016

trioni commented Mar 31, 2016

Dbuggerx commented Jun 6, 2016

ocrunch commented Jun 8, 2016

slimatic commented Jun 9, 2016

xinbin commented Jun 14, 2016 •

edited

bjorndavis commented Jul 6, 2016

catchergeese commented Jul 11, 2016

kyleskrinak commented Aug 29, 2016

ChrisBAshton commented Aug 29, 2016 •

edited

Peter-Petrik commented Oct 11, 2016 •

edited

sembrat commented May 8, 2018

edurenye commented Nov 30, 2018

ErroneousBosch commented Mar 13, 2019

spider_skips property broken #401

spider_skips property broken #401

Comments

paullew commented Mar 14, 2016

altV commented Mar 18, 2016

imagreenplant commented Mar 27, 2016

Vexrm commented Mar 28, 2016

trioni commented Mar 31, 2016

Dbuggerx commented Jun 6, 2016

ocrunch commented Jun 8, 2016

slimatic commented Jun 9, 2016

xinbin commented Jun 14, 2016 • edited

bjorndavis commented Jul 6, 2016

catchergeese commented Jul 11, 2016

kyleskrinak commented Aug 29, 2016

ChrisBAshton commented Aug 29, 2016 • edited

Peter-Petrik commented Oct 11, 2016 • edited

sembrat commented May 8, 2018

edurenye commented Nov 30, 2018

ErroneousBosch commented Mar 13, 2019

`spider_skips` property broken #401

`spider_skips` property broken #401

xinbin commented Jun 14, 2016 •

edited

ChrisBAshton commented Aug 29, 2016 •

edited

Peter-Petrik commented Oct 11, 2016 •

edited