Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[NAT-3.9] Less insane NAT scraper #497

Merged
merged 12 commits into from
May 17, 2023
Merged

[NAT-3.9] Less insane NAT scraper #497

merged 12 commits into from
May 17, 2023

Conversation

CommanderStorm
Copy link
Member

During #495 I noticed that there are still some pretty insane shenanigans going on in the scraped results for the nat

Proposed Changes (include Screenshots if possible)

  • Made the Orgs an external thing (thus merging the org vs. orgs field dilema)
  • Made sure that all translated fields are properly registered

How to test this PR

  • Code Review

How has this been tested?

  • Internal monolouge
  • Letting it run

Checklist:

  • I have updated the documentation / No need to update the documentation
  • I have run the linter

@CommanderStorm CommanderStorm self-assigned this Apr 3, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Apr 3, 2023

👋 Thank you for contributing. A staging environment for this PR for this change will be available shortly

@CommanderStorm CommanderStorm changed the title made shure that the nat scraper is a bit more sane Less insane NAT scraper Apr 3, 2023
github-actions bot added a commit that referenced this pull request Apr 3, 2023
@github-actions
Add deployment for PR #497
255df4a
octycs
octycs reviewed Apr 3, 2023
View reviewed changes
data/external/scrapers/nat.py Outdated Show resolved Hide resolved
data/external/scrapers/nat.py Outdated Show resolved Hide resolved
data/external/scrapers/nat.py Outdated Show resolved Hide resolved
data/external/scrapers/nat.py Outdated Show resolved Hide resolved
@CommanderStorm
Copy link
Member Author

as via email I have removed the support for:

  • orgs
  • corona
  • bauarbeiten
  • contact

This also adds support for maps

@CommanderStorm CommanderStorm requested a review from octycs May 7, 2023 12:53
@CommanderStorm CommanderStorm changed the title Less insane NAT scraper [NAT-3.9] Less insane NAT scraper May 7, 2023
@CommanderStorm CommanderStorm added the data This issue is related to the data collection and aggregation label May 16, 2023
octycs
octycs approved these changes May 16, 2023
View reviewed changes
Copy link
Contributor

@octycs octycs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just added more information in the docstrings of the translation functions.

data/external/scrapers/nat.py Outdated Show resolved Hide resolved
data/external/scrapers/nat.py Show resolved Hide resolved
@CommanderStorm CommanderStorm enabled auto-merge (squash) May 17, 2023 13:53
@CommanderStorm @octycs
Apply suggestions from code review
859e427 
Co-authored-by: octycs <octycs@users.noreply.github.com>
@CommanderStorm CommanderStorm merged commit aaf51ad into main May 17, 2023
@CommanderStorm CommanderStorm deleted the sane_nat_scraper branch May 17, 2023 15:14
github-actions bot added a commit that referenced this pull request May 17, 2023
@github-actions
Removed the deployment for PR #497
80654da
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@octycs octycs octycs approved these changes

Assignees

@CommanderStorm CommanderStorm

Labels
data This issue is related to the data collection and aggregation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@CommanderStorm @octycs