Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adiciona spider base para o sistema replicável Atende, e spiders derivados para as cidades com D.O.s nesse sistema. #1046

Draft
wants to merge 40 commits into
base: main
Choose a base branch
from
Draft

Adiciona spider base para o sistema replicável Atende, e spiders derivados para as cidades com D.O.s nesse sistema. #1046

wants to merge 40 commits into from

Commits on Dec 2, 2023

  1. Spider for Campo Mourao/PR

    @rodps @AlexJBSilva
    rodps authored and AlexJBSilva committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    b178d13 View commit details
    Browse the repository at this point in the history

  2. automated code formatting

    @AlexJBSilva
    allison authored and AlexJBSilva committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    efe5f05 View commit details
    Browse the repository at this point in the history

  3. Changes in Campo Mourao spider

    @rodps @AlexJBSilva
    rodps authored and AlexJBSilva committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    5ebd2ed View commit details
    Browse the repository at this point in the history

  4. Move pr_campo_mourao criado em okfn-brasil#438 para servir de referên… 

    …cia na criação do spider base do sistema replicável Atende.
    @AlexJBSilva
    AlexJBSilva committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    0b8a210 View commit details
    Browse the repository at this point in the history

  5. Adiciona spider base do sistema replicável Atende. 

    Essa versão implementa a classe 'BaseAtendeT2Spider' para buscar os
dirários nas páginas com layout 'Tipo 2', identificadas pelo Mapeador
Atende desenvolvido em okfn-brasil#1043.
    @AlexJBSilva
    AlexJBSilva committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    88f036c View commit details
    Browse the repository at this point in the history

  6. Ajusta o spider 'pr_campo_mourao' criado em okfn-brasil#438 

    para trabalhar com o spider base do sistema replicável 'Atende'.
Resolve okfn-brasil#430
Adiciona spider para Campo Mourão - PR.
    @AlexJBSilva
    AlexJBSilva committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    5158d72 View commit details
    Browse the repository at this point in the history

  7. Ajusta o spider 'rs_gravatai' para trabalhar com o 

    spider base do sistema replicável 'Atende'.
    @AlexJBSilva
    AlexJBSilva committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    c463564 View commit details
    Browse the repository at this point in the history

Commits on Dec 3, 2023

  1. Adiciona o spider 'pr_araucaria' de Araucária PR. 

    Sistema replicável 'Atende'.
    @AlexJBSilva
    AlexJBSilva committed Dec 3, 2023
    Configuration menu
    Copy the full SHA
    06e517f View commit details
    Browse the repository at this point in the history

  2. Ajuste no spider base 'atende' para buscar a url 

    de download na página de 'Detalhes da Edição' quando não encontrar na
página padrão (layout Tipo 2).
Caso conhecido: Araucária - PR (https://araucaria.atende.net/diariooficial)
    @AlexJBSilva
    AlexJBSilva committed Dec 3, 2023
    Configuration menu
    Copy the full SHA
    8aa4867 View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2023

  1. Adiciona o spider 'pr_apucarana' de Apucarana - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    443b256 View commit details
    Browse the repository at this point in the history

  2. Adiciona o spider 'sc_araquari' de Araquari - SC. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    1e709d9 View commit details
    Browse the repository at this point in the history

  3. Adiciona o spider 'rs_bento_goncalves' de Bento Gonçalves - RS. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    79d3e6e View commit details
    Browse the repository at this point in the history

  4. Adiciona o spider 'pr_campo_largo' de Campo Largo - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    9e28719 View commit details
    Browse the repository at this point in the history

  5. Adiciona o spider 'rs_candelaria' de Candelária - RS. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    b5b88fc View commit details
    Browse the repository at this point in the history

  6. Adiciona o spider 'mg_carmopolis_de_minas' de Carmópolis de Minas - MG. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    d5563a2 View commit details
    Browse the repository at this point in the history

  7. Adiciona o spider 'pr_castro' de Castro - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    7320399 View commit details
    Browse the repository at this point in the history

  8. Adiciona o spider 'pr_clevelandia' de Clevelândia - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    544aaee View commit details
    Browse the repository at this point in the history

  9. Adiciona o spider 'pr_corbelia' de Corbélia - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    1cacfd8 View commit details
    Browse the repository at this point in the history

  10. Adiciona o spider 'rs_dois_irmaos' de Dois Irmãos - RS. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    240da15 View commit details
    Browse the repository at this point in the history

  11. Adiciona o spider 'rs_estrela' de Estrela - RS. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    b56b8c3 View commit details
    Browse the repository at this point in the history

  12. Adiciona o spider 'pr_guaraniacu' de Guaraniaçu - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    a7c1884 View commit details
    Browse the repository at this point in the history

  13. Adiciona o spider 'rs_horizontina' de Horizontina - RS. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    31a2cf2 View commit details
    Browse the repository at this point in the history

  14. Adiciona o spider 'pr_juranda' de Juranda - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    f5c3000 View commit details
    Browse the repository at this point in the history

  15. Adiciona o spider 'sc_laurentino' de Laurentino - SC. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    de9f280 View commit details
    Browse the repository at this point in the history

  16. Adiciona o spider 'pr_mambore' de Mamborê - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    59a3565 View commit details
    Browse the repository at this point in the history

  17. Adiciona o spider 'mg_oliveira' de Oliveira - MG. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    238dbef View commit details
    Browse the repository at this point in the history

  18. Adiciona o spider 'pr_ouro_verde_do_oeste' de Ouro Verde do Oeste - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    1b56af0 View commit details
    Browse the repository at this point in the history

  19. Adiciona o spider 'rs_panambi' de Panambi - RS. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    b1c76b6 View commit details
    Browse the repository at this point in the history

  20. Adiciona o spider 'pr_pinhais' de Pinhais - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    a3f834a View commit details
    Browse the repository at this point in the history

  21. Adiciona o spider 'pr_rio_branco_do_sul' de Rio Branco do Sul - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    5bedc3c View commit details
    Browse the repository at this point in the history

  22. Adiciona o spider 'rs_santa_rosa' de Santa Rosa - RS. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    7a04b86 View commit details
    Browse the repository at this point in the history

  23. Adiciona o spider 'pr_santo_antonio_da_platina' de Santo Antônio da P… 

    …latina - PR.

Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    93a47bc View commit details
    Browse the repository at this point in the history

  24. Adiciona o spider 'rs_sao_joao_do_polesine' de São João do Polêsine -… 

    … RS.

Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    2ed19e3 View commit details
    Browse the repository at this point in the history

  25. Adiciona o spider 'rs_sobradinho' de Sobradinho - RS. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    9ad4d0c View commit details
    Browse the repository at this point in the history

  26. Adiciona o spider 'pr_tupassi' de Tupãssi - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    8479713 View commit details
    Browse the repository at this point in the history

  27. Adiciona o spider 'pr_pato_bragado' de Pato Bragado - PR. 

    Sistema replicável 'Atende' (layout Tipo 2).
    @AlexJBSilva
    AlexJBSilva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    cf144d7 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2023

  1. Adiciona a classe 'BaseAtendeT1Spider' para buscar 

    os diários nas páginas do sistema replicável 'atende' com layout 'Tipo 1'.
    @AlexJBSilva
    AlexJBSilva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    9d362ff View commit details
    Browse the repository at this point in the history

  2. Adiciona o spider 'rs_cachoeirinha' de Cachoeirinha - RS. 

    Sistema replicável 'Atende' (layout Tipo 1).
    @AlexJBSilva
    AlexJBSilva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    6e952ce View commit details
    Browse the repository at this point in the history

  3. Adiciona o spider 'rs_camaqua_2023' de Camaquã - RS. 

    Sistema replicável 'Atende' (layout Tipo 1).
Resolve okfn-brasil#1038
Conforme comentado em okfn-brasil#1038 (comment)
esse novo spider coleta apenas da edição 333 em diante.
Como falta a informação de data para muitas edições na página,
a estratégia foi incluir a verificação do número da edição para limitar
o download dos arquivos e salvar a data
`datetime.date.max` nos metadados para que a etapa de 'data processing'
realize a extração dessa informação.
    @AlexJBSilva
    AlexJBSilva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    a944881 View commit details
    Browse the repository at this point in the history

  4. Inclui a informação de 'end_date' para o spider 

    original de Camaquã, para refletir o último dia antes da publicação da
edição 333 no novo site atende.net
Ajuste associado à issue okfn-brasil#1038
    @AlexJBSilva
    AlexJBSilva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    ded3a1e View commit details
    Browse the repository at this point in the history