Replies: 4 comments 13 replies
|
Even if we just write the html, the text extraction in warc2text is required for the tag filters that remove google translated wordpress documents. Are you ok doing that kind of filtering in stage 2? It's just a matter of detecting certain html tags and could be easily done with xpath. |
|
Regarding the output format, at Prompsit we are discarding any use of binary indexed formats, since it will complicate things in warc2text to much more development time than we want to expend before starting the extraction. So then, we have two options left: all the {meta}data in jsonl or only metadata in jsol and html in b64. Despite I agree with the fact that is very easy to perform metadata-related queries when it is separated, I think it will complicate things much more for the stage 2 parallelization. If we run, let's say, 256 jobs of warc2text, we will have 256 jsonl file + 256 b64 files. Given that stage 2 will need more parallelization than running 256 python processes, parallelization within each file using GNU parallel would be needed. But I don't think that's an easy solution if each document has to be read from two sources (metadata, html). When for example, with a single jsonl file it would be as easy as: |
|
All the changes that we've been talking about are made now in #50. But I made a couple of changes more, in separated PRs because we may not merge them. These changes are:
I made these two new changes because we probably may need to cut down the output size significantly. As we talked about in a discussion in Mattermost, we don't have much space on NIRD and the extraction of HTML instead of text decreases much less the size of the WARCS. About 1/3 or 1/2 compared to 10x-30x times as before extracting text. As an example, 50GB of random WARCs yielded 16GB of html.gz and 2.9GB of text.gz. With the changes above mentioned, taking advantage of ZSTD compression and the JSON plain text format we can obtain much higher compression ratios (as we early discussed b64 is compressing less than plain text). You may notice that the changes are made in form of a combination of pluggable features. This is because I wanted to include and maintain the changes in the main repo and not having them splitted between main and fork. So I tried to make all of them in a way they don't break the internal design and don't change the default behaviour, but let us in HPLT manage the output format more or less as we need. EDIT: Please if you want to try all the changes, use bitextor/warc2text#52. As the PR are made incrementally and that one includes all the features. |
|
A new release has been made with the changes. If there is nothing more needed, I guess processing can start. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
There has been extensive discussion between the Prompsit and UiO teams about the technical setup for our second round of data preparation. We will summarize and collectively discuss here, aiming for decisions by the end of January.
Functionality Overview: Stage 1 (Currently
warc2text)robots.txt)robots.txtto new archiveNew Tool: Stage 2
To filter based on
robots.txt-compliance, we need to insert an additional step: Stage 1 collects allrobots.txtacross WARC files and crawls; to flag (or remove) documents based on this, we need to compare all URLs against this new mega-index. This applies to mono- and bi-texting alike and may substantially remove the overall volume of data, so should be applied prior to downstream processing.Unrelated to the above, we are tempted to put more functionality into text extraction, notably boilerplate detection (or removal), e.g. using a tool like Trafilatura. Doing this prior to (document-level) language identification, will likely lead to more consistent language identification. Therefore, this would also have to go into Stage 2 then.
A major benefit of breaking text extraction into two stages is that Stage 1 can be lean and cheap, so that it can be run directly on the storage nodes. Record selection from the WARC files (skipping e.g. binary files, images, videos, non-HTML documents) massively reduces the data volume, so that the resulting input to Stage 2 is a lot easier to transfer to a compute cluster. Running Stage 1 on a real cluster, in turn, means we can be more ambitious in processing, e.g. afford full HTML DOM parsing to support boilerplate detection (and possibly better segmentation). It also means we can (probably) implement Stage 2 in Python (rather than in C++), where HPLT has a larger developer team available.
For this scenario, Stage 1 (
warc2text) needs to be refined (into something that could be calledwarc2html) to write out complete HTML documents, while skipping text extraction, segmentation, and language identification (also, it appears that encoding normalization currently is only applied during text extraction, so it would have to be moved slightly earlier insidewarc2text). Abstractly, its output should be complete HTML documents and associated metadata.In this revised scenario, we are free to define Stage 1 output independent of user existing use cases for
warc2text; for now at least, this mode would only be used as part of the revised HPLT text extraction pipeline. Two options have been discussed:(1) A conservative choice would be to moderately adjust existing output code in
warc2textand keep the HTML documents separate from the metadata, i.e. create pairs of line-aligned files: one with BASE64-encoded HTML documents, the other with all metadata in JSONlines. The rationale for the separation is volume. HTML documents likely account for the by far biggest part of the data; being able to just process metadata (e.g. for analytics, orrobots.txt-based filtering) without having to read through all the textual content may be beneficial. On the other hand, integrating all metadata into a single JSONlines file will make the pipeline more HPC-friendly and practically make it easier to deal with groups of metadata fields.(2)
All reactions