Unable to pass the url

[Gautamshahi]

r = requests.get(url)
data = extruct.extract(r.text, r.url)

why i am getting error in this way?

[redapple]

What error are you getting @Gautamshahi ? Please share your stack trace.
What version of extruct are you using?

[Gautamshahi]

I am using extruct 0.4.0.

In this screenshot you can see the error and the code, actually i am tring to pass the URL from the warc file.

[redapple]

This does not look related to extruct. It has more to do with warc's record headers and how to get the URL from a record.
I'm not familiar with warc though, so I cannot help you.

[redapple]

Also, in the loop, you are using r.text and r.url to pass HTML and URL to extruct.
But you are reading records from a warc file. You'll need to adapt the example from extruct's README to get HTML and URL from record in your loop.

[pmyteh]

This is a warc error rather than an extruct one. The problem is, I think, that not all records in a WARC archive actually contain crawled data. Depending on your WARC source, some contain HTTP requests, and some also contain 'warcinfo' metadata. A (late draft) copy of the WARC standard can be found at the BNF.

If you iterate only over 'response' WARC records you should find it works out. But in any case, this isn't an extruct error.

[Gautamshahi]

I am not expert both in EXTRUCT and WARC but I check the WARC code ( record.header['WARC-Target-URI']) it fetches the list of URL the warc file.

Moreover, if I just save list of URL in a text file(after extracting from WARC file) example-
http://01.rknt.jp/frienzu/
http://01my.blogspot.com/2015/06/
http://03188b9.netsolhost.com/fragorder.html
http://036.asiapars.com/
http://0avocat.ca/portal/fr/ArticleView.php?article_id=69
http://0lt3den3ox.jugem.jp/?eid=26

Can I pass the URL one by one from text file to the code?
r = requests.get(url)
data = extruct.extract(r.text, r.url)

If so, can you suggest the possible method?

I hope its clear to you now.

[Gautamshahi]

I am not expert both in EXTRUCT and WARC but I check the WARC code ( record.header['WARC-Target-URI']) it fetches the list of URL the warc file. Moreover, if I just save list of URL in a text file(after extracting from WARC file) example- http://01.rknt.jp/frienzu/ http://01my.blogspot.com/2015/06/ http://03188b9.netsolhost.com/fragorder.html http://036.asiapars.com/ http://0avocat.ca/portal/fr/ArticleView.php?article_id=69 http://0lt3den3ox.jugem.jp/?eid=26 Can I pass the URL one by one from text file to the code? r = requests.get(url) data = extruct.extract(r.text, r.url) If so, can you suggest the possible method? I hope its clear to you now.

…

On Thu, Aug 31, 2017 at 3:11 PM, Paul Tremberth ***@***.***> wrote: Also, in the loop, you are using r.text and r.url to pass HTML and URL to extruct. But you are reading records from a warc file. You'll need to adapt the example from extruct's README to get HTML and URL from record in your loop. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#54 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADyFFkWP6hK71hom8E7jUzY_ApdNIBCaks5sdn_XgaJpZM4PIJ5C> .

-- Gautam Kishore Shahi, Master Student, DISI- University of Trento, Italy

[redapple]

@Gautamshahi , you can try that yest, it should work fine.
Though if you have the WARC file already, you probably have HTTP response bodies already.
Downloading the URLs again with requests is doing the work again.
It's your call.

[pmyteh]

You can only extruct what's actually there. Without knowing exactly what's in your WARC file it's difficult to answer your question.

Something like:

for record in f:
    if record.type != 'response':
        continue
    <do extruct stuff on record.url and record.text>

Should help.

I'm happy to help more, but as this is not an extruct problem I don't want to do it here. Drop me a line direct if you need it.