Scraping several sites at the same time

[racindustries]

When running the code only the first news website entered in the json list seems to be downloaded and parsed. Do you have any suggestion ?

[ghost]

@racindustries same with me. And I've looked around to see if anyone has a solution. Haven't found any.
Were you able to work around this issue?

[Susmithap3]

same i also need help???

[ivanovishado]

Can any of you please share the code that you're using?
I used the code of this repo and worked fine with the JSON list I provided to it.

[ghost]

It's been a while from my end, but i used the exact same code from holwech only changed the news sites.

…

On Mon, Oct 22, 2018 at 9:40 AM Iván Galaviz ***@***.***> wrote: Can any of you please share the code that you're using? I used the code of this repo and worked fine with the JSON list I provided to it. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AlG3evwRtK9o_L2zmixOLc9R8RN5RJZzks5unWhIgaJpZM4Vi9hr> .

[ivanovishado]

@Civmwa can you please share the JSON list you used to see if I can reproduce the error?

[ghost]

@ivanovishado
{
"The Standard": {
"link": "https://www.standardmedia.co.ke/business"
},
"bbc": {
"rss": "http://feeds.bbci.co.uk/news/rss.xml",
"link": "http://www.bbc.com/"
},
"theguardian": {
"rss": "https://www.theguardian.com/uk/rss",
"link": "https://www.theguardian.com/international"
},
"breitbart": {
"link": "http://www.breitbart.com/"
},
"infowars": {
"link": "https://www.infowars.com/"
},
"foxnews": {
"link": "http://www.foxnews.com/"
},
"nbcnews": {
"link": "http://www.nbcnews.com/"
},
"washingtonpost": {
"rss": "http://feeds.washingtonpost.com/rss/world",
"link": "https://www.washingtonpost.com/"
}
}

[ivanovishado]

@Civmwa NewsScraper.py worked fine for me, here's the output file as proof.

Tested it in Windows 10, Python 3.6.2

[ghost]

Hi Ivan - Not entirely sure what happened between when i sent it to you and now, but i ran it and it works. LOL. One small issue though, how would i get to print a summary of the article?

…

On Thu, Oct 25, 2018 at 7:22 AM Iván Galaviz ***@***.***> wrote: @Civmwa <https://github.com/Civmwa> NewsScraper.py worked fine for me, here's the output file <https://pastebin.com/ndLPb7QL> as proof. Tested it in Windows 10, Python 3.6.2 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AlG3etznaAb8qfydiRykeT8q-zZc6P27ks5uoTyHgaJpZM4Vi9hr> .

[ivanovishado]

Not entirely sure what happened between when i sent it to you and
now, but i ran it and it works. LOL.

@Civmwa lol

how would i
get to print a summary of the article?

You need to add content.nlp() just after content.parse() then you would call content.summary.
Keep in mind that nlp() adds some processing time and the summary won't be perfect.

[ghost]

Thanks Ivan. Much appreciated

…

On Fri, Oct 26, 2018 at 6:21 AM Iván Galaviz ***@***.***> wrote: Not entirely sure what happened between when i sent it to you and now, but i ran it and it works. LOL. @Civmwa <https://github.com/Civmwa> lol how would i get to print a summary of the article? You need to add content.nlp() just after content.parse() then you would call content.summary. Keep in mind that nlp() adds some processing time and the summary won't be perfect. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AlG3ejf3IiyYn8ghMi3lXYVRf3nHneZFks5uon-ogaJpZM4Vi9hr> .

[ivanovishado]

@Civmwa You're welcome.
I believe this issue can be closed now, @racindustries.

[ghost]

Yes.

…

On Mon, Oct 29, 2018 at 4:46 AM Iván Galaviz ***@***.***> wrote: @Civmwa <https://github.com/Civmwa> You're welcome. I believe this issue can be closed now, @racindustries <https://github.com/racindustries>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AlG3eutQdj1alQiwLXzLmkJUsIkRHN3nks5upl4NgaJpZM4Vi9hr> .