Fecam #135

jvanz · 2019-11-12T11:35:57Z

This is the initial version of a spider to get the data from FECAM website. I've created a spider class which able to understand the fecam website and navigate thou the pages and extract the gazette file.

Most of the file are docs. Thus, I've also added a new step in the pipeline to convert the doc into pdf. For that, i used Libreoffice writer. Its the best option in the top of my mind, if you have a better idea, let me know.

I'll continue testing and adding the missing cities which have their gazette in the website. Currently, I'm just testing with my city

The files are being downloaded from: https://diariomunicipal.sc.gov.br/site/


        FECAM spider

Adds a spider to crawl the data of the cities which are associate to FECAM Issue #134 Signed-off-by: José Guilherme Vanz <jvanz@jvanz.com>


        DOC to PDF pipeline

Adds a step in the pipeline to convert doc[x] file to pdf. Some gazette files from SC cities are available in doc formats. Issue #134 Signed-off-by: José Guilherme Vanz <jvanz@jvanz.com>

vitorbaptista · 2019-11-12T13:03:28Z

Hello @jvanz! Nice PR. Just so I understand, why are you converting a DOC to a PDF? Is it just so we can reuse the PDF parsing pipeline?

I'm not from the core team, but it is a strange strategy to convert from a "more machine-readable" format (doc) to a less readable format (pdf) before parsing the data. IMHO, it would be better to change the pipeline to support not only PDFs, but also DOCs. Especially because I imagine there will be other formats in the future (e.g. TXT?), and this current requirement that the gazettes are PDFs will have to be changed anyway.

I imagine the PdfParsingPipeline becoming something like a TextExtractorPipeline, and supporting PDFs, DOC, etc. Have you tried something like this?

jvanz · 2019-11-12T13:13:26Z

Hello @jvanz! Nice PR. Just so I understand, why are you converting a DOC to a PDF? Is it just so we can reuse the PDF parsing pipeline?

Yes

I'm not from the core team, but it is a strange strategy to convert from a "more machine-readable" format (doc) to a less readable format (pdf) before parsing the data. IMHO, it would be better to change the pipeline to support not only PDFs, but also DOCs. Especially because I imagine there will be other formats in the future (e.g. TXT?), and this current requirement that the gazettes are PDFs will have to be changed anyway.

I imagine the PdfParsingPipeline becoming something like a TextExtractorPipeline, and supporting PDFs, DOC, etc. Have you tried something like this?

I decided to not extract the txt from the doc because I didn't find a good lib for that. I know there is the python-docx library. But from previous experiences, where it was not able to read all docish files, I decided to convert to pdf. However, if you think that the way to go, I can update the PR. :)

vitorbaptista · 2019-11-12T13:42:14Z

@jvanz Regarding extracting text from DOC, have you tried Apache Tika? I've never used it, but heard good things. If it works well, I think it's the ideal tool, as it supports dozens of file formats (including PDFs). What do you think, @cuducos / @Irio ?

jvanz · 2019-11-12T13:45:31Z

@jvanz Regarding extracting text from DOC, have you tried Apache Tika? I've never used it, but heard good things. If it works well, I think it's the ideal tool, as it supports dozens of file formats (including PDFs). What do you think, @cuducos / @Irio ?

I never have tried Apache Tika. Actually, I never have hear about it. It seems interesting indeed

processing/data_collection/gazette/spiders/base.py

processing/data_collection/gazette/pipelines.py

cuducos · 2019-11-12T14:02:55Z

Many thanks for the contribution, @jvanz! Really great code and solutions IMHO.

I don't know Apache Tika either, but I have the feeling that it would be more trustworthy to extract text from Word document than from PDF – I mean, the quality of the extracted text would be considerably better coming from .doc than from .pdf. But it's only a guess.

In other ways, I would consider extracting the text from the Word document and not from the converted PDF.


        pipelines.py: lower file path

To ensure that the function is_doc work with all file paths, this commit adds a lower() call before the test. Thus, file path in upper case will work as well. Signed-off-by: José Guilherme Vanz <jvanz@jvanz.com>


        base.py: fix import order

To be compliance with PEP8, this commit changes the order of the import statements. Signed-off-by: José Guilherme Vanz <jvanz@jvanz.com>


        pipelines.py: do not remove doc files

To keep track of which file originates the PDF files remove the os.unlink call used to delete the doc files. Furthermore, keep the doc file extension in the pdf file path. Signed-off-by: José Guilherme Vanz <jvanz@jvanz.com>

jvanz · 2019-11-12T21:43:06Z

I believe that all issues are solved in the last commits. Related to the Apache Tika, do you agree to leave this for a separate PR? I can work on that before adding other spiders for other cities. I can open an issue for that an keep track there.


        pipelines.py: using Apache Tika to get doc content

Add a class in the pipeline able to detect the file type and extract the text using the best tool. The current implementation uses the already in-place tool for pdf files. For docish files, the new file type added, the pipeline step uses Apache Tika to get the content. #136 Signed-off-by: José Guilherme Vanz <jvanz@jvanz.com>

processing/data_collection/gazette/pipelines.py

vitorbaptista · 2019-11-15T13:46:34Z

Apart from the small comment I added inline, this is looking great! I haven't tried running it, but it seems ready to go 👍


        pipelines.py: add ".txt" extension

Keep the pdf file extension in the txt file path. Thus, it's more easy to keep track which file originates the txt file Signed-off-by: José Guilherme Vanz <jvanz@jvanz.com>


        sc_gaspar.py: removes unused imports

The ScGasparSpider just need to import the FecamGazetteSpider spider. This commit removes all the other imports Signed-off-by: José Guilherme Vanz <jvanz@jvanz.com>


        pipelines.py: set the text path file

In the pipeline item to extract the text from the pdf file, when building the command string adds the path to the text file path as well. Thus, we avoid "No such file or directory" errors Signed-off-by: José Guilherme Vanz <jvanz@jvanz.com>

jvanz · 2019-11-27T01:24:08Z

So guys, what do you think? Is the PR in the right path? :)


        pipelines.py: fix type error

Fixes a type error raising an Exception object instead of the error string. Signed-off-by: José Guilherme Vanz <jvanz@jvanz.com>

processing/data_collection/gazette/spiders/base.py

vitorbaptista

I haven't executed the code, but it looks very good! Just found a tiny typo, and that's good to go 👍


        base.py: fix typo

Fixes a typo in the method name Signed-off-by: José Guilherme Vanz <jvanz@jvanz.com>

jvanz added 2 commits Nov 10, 2019

cuducos reviewed Nov 12, 2019

View changes

processing/data_collection/gazette/spiders/base.py Show resolved Hide resolved

cuducos reviewed Nov 12, 2019

View changes

processing/data_collection/gazette/pipelines.py Show resolved Hide resolved

cuducos reviewed Nov 12, 2019

View changes

processing/data_collection/gazette/pipelines.py Outdated Show resolved Hide resolved

cuducos reviewed Nov 12, 2019

View changes

processing/data_collection/gazette/pipelines.py Outdated Show resolved Hide resolved

jvanz added 3 commits Nov 12, 2019

jvanz requested a review from cuducos Nov 13, 2019

jvanz mentioned this pull request Nov 13, 2019

Apache Tika #136

Closed

vitorbaptista reviewed Nov 15, 2019

View changes

processing/data_collection/gazette/pipelines.py Outdated Show resolved Hide resolved

jvanz force-pushed the jvanz:fecam branch from 050659a to b0442a7 Nov 15, 2019

jvanz requested a review from vitorbaptista Nov 17, 2019

jvanz force-pushed the jvanz:fecam branch from 3f03268 to 34cfea4 Nov 17, 2019

vitorbaptista reviewed Nov 29, 2019

View changes

processing/data_collection/gazette/spiders/base.py Outdated Show resolved Hide resolved

vitorbaptista approved these changes Nov 29, 2019

View changes

base.py: fix typo

Verified

This commit was signed with a verified signature.

jvanz José Guilherme Vanz

GPG key ID: 4159E08B20565EF1 Learn about signing commits

Loading status checks…

35072c7

Fixes a typo in the method name Signed-off-by: José Guilherme Vanz <jvanz@jvanz.com>

sergiomario merged commit d84552a into okfn-brasil:master Nov 29, 2019
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

okfn-brasil/diario-oficial

Fecam #135

Fecam #135

jvanz commented Nov 12, 2019 •

edited

This comment has been minimized.

vitorbaptista commented Nov 12, 2019

This comment has been minimized.

jvanz commented Nov 12, 2019 •

edited

This comment has been minimized.

vitorbaptista commented Nov 12, 2019

This comment has been minimized.

jvanz commented Nov 12, 2019

This comment has been minimized.

cuducos commented Nov 12, 2019

This comment has been minimized.

jvanz commented Nov 12, 2019 •

edited

This comment has been minimized.

vitorbaptista commented Nov 15, 2019

This comment has been minimized.

jvanz commented Nov 27, 2019

vitorbaptista left a comment

okfn-brasil/diario-oficial

Join GitHub today

Fecam #135

Conversation

jvanz commented Nov 12, 2019 • edited

This comment has been minimized.

vitorbaptista commented Nov 12, 2019

This comment has been minimized.

jvanz commented Nov 12, 2019 • edited

This comment has been minimized.

vitorbaptista commented Nov 12, 2019

This comment has been minimized.

jvanz commented Nov 12, 2019

This comment has been minimized.

cuducos commented Nov 12, 2019

This comment has been minimized.

jvanz commented Nov 12, 2019 • edited

This comment has been minimized.

vitorbaptista commented Nov 15, 2019

This comment has been minimized.

jvanz commented Nov 27, 2019

vitorbaptista left a comment

jvanz commented Nov 12, 2019 •

edited

jvanz commented Nov 12, 2019 •

edited

jvanz commented Nov 12, 2019 •

edited