Question: how to store data in json

[SophiaCY]

Hi, recently I have installed portia and it's quite good. But since I'm not very familiar with it, I have met a problem:
In my template I used variants to scrape the website and it worked well when I clicked continue browsing or turned to a similiar page. However, when I started running the portia spider in cmd, it could successfully scrape data yet failed to store the whole data scraped in the json file. I found that in the json file only the data in the page I annotated were stored.
I guess the problem is "WARNING: Dropped: Duplicate product scraped at http://bbs.chinadaily.com.cn/forum-83-206.html, first one was scraped at http://bbs.chinadaily.com.cn/forum-83-2.html" (it appeared in cmd when the spider was running), but I don't know how to solve the problem.
I hope someone can help me as soon as possible.
Thanks very much!

[ruairif]

For portiacrawl if you add -o FILENAME.json items will be saved to the file you specify.
Running portiacrawl -h gives more info.

[SophiaCY]

Sorry, maybe I didn't express my problem clearly.
I want to scrape data in chinadaily's forum, so I annotate the page http://bbs.chinadaily.com.cn/forum-83-2.html and in crawling, I set the follow pattern as http://bbs.chinadaily.com.cn/forum-83-\d+.html

In my template I use variants to scrape items:

and it works well when I click continue browsing or turn to a similiar page:

So I start to run the spider in console:
cd C:\Users\Administrator\portia\portia\slybot\slybot
portiacrawl C:\Users\Administrator\portia\portia\slyd\data\projects\new_project bbs.chinadaily.com.cn -o itemsss.json -t json

Then I can see the data I want in my console, but when the spider finished, only the items in http://bbs.chinadaily.com.cn/forum-83-2.html are stored:

I don't know what I'm missing...

Thanks very much!!!;)

[ruairif]

What are your start urls?

[SophiaCY]

http://bbs.chinadaily.com.cn/forum-83-2.html
and it's also the page I annotate.

[ruairif]

Which links are green when you check the 'Overlay blocked links' box?

[SophiaCY]

Just the links shown in the picture:

[ruairif]

Does it work if you create a file slybot/local_slybot_settings.py with the following content?:

ITEM_PIPELINES = {'slybot.dupefilter.DupeFilterPipeline': None}

[SophiaCY]

Yes it works when I store data in a json file. Thank you very much!!!;)
However, because the data in the json file can't be imported to the database directly, I have to store data in a csv file. But when I run the spider with:
portiacrawl C:\Users\Administrator\portia\portia\slyd\data\projects\new_project bbs.chinadaily.com.cn -t csv -o p.csv
the spider runs successfully but only the names of the items are stored in the csv file.

Thanks a lot!

[ruairif]

I'm not sure what's happening with your CSV file but with your JSON file there are a few other things you could try that might make it more suitable to being inserted into the database.

The duplicate filter was being triggered because all of your fields are marked as variant and as such only the first item is being kept. You could solve this by adding a dummy non variant field that is different on each page.
Another option you could try would be to use the Split Variants Middleware to create individual items for each entry. I'm not sure if these instructions are exactly correct but it would be great if you could try it out:

pip install scrapylib

Then in your local_slybot_settings.py file add:

SPIDER_MIDDLEWARES = {'slybot.spiderlets.SpiderletsMiddleware': 998,
                      'scrapylib.splitvariants.SplitVariantsMiddleware': 999}
SPLITVARIANTS_ENABLED = True

Please report back if it goes well or if you need more help.

[SophiaCY]

Thanks a lot!
I have tried your second method and it successfully created individual items for each entry.
Here is the result:
the former:

the latter:

By the way, then I tried to store data in the csv file and they're successfully stored! Maybe it's your second method that helped me solve the problem.
It's very nice of you! Thank you very much!

[ruairif]

You're very welcome.
Thank you for testing it I'll add that info to the README next time I update it.

[SophiaCY]

My pleasure ^_^

[maoouyang]

how to store in mysql ? I am puzzle！！

[maoouyang]

I am first use of portia,I had store data to json,but I dont kown how to store in mysql ..thanks.

[ghost]

same problem? hello maoouyang have you understand how to do?

[AlexTan-b-z]

I set same with you,but I alwals received :
2016-08-25 16:03:37 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
why?
and the json file is empty,only have [.
who can help me? thank you!

[AlexTan-b-z]

Now,my question is same with you.
Json only the first page are stored.
CSV only the names of the items.
However I use the both two methods,it doesn't work.
How can I do it?