Skip to content

Improve web page crawler #795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 19, 2023
Merged

Improve web page crawler #795

merged 1 commit into from
Sep 19, 2023

Conversation

m-1-k-3
Copy link
Member

@m-1-k-3 m-1-k-3 commented Sep 19, 2023

  • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

Feature

  • What is the current behavior? (You can also link to an open issue here)

On some web servers we get 200 ok for pages that are not available

  • What is the new behavior (if this is a feature change)? If possible add a screenshot.

We do a dummy random request to identify a not available response and use this to compare to the further responses.

@m-1-k-3 m-1-k-3 added enhancement New feature or request System emulator (Lxx) The system emulator (Lxx modules) EMBA labels Sep 19, 2023
@m-1-k-3 m-1-k-3 self-assigned this Sep 19, 2023
@m-1-k-3
Copy link
Member Author

m-1-k-3 commented Sep 19, 2023

On my testfirmware we improved the results massively:

Before:
16726 /home/m1k3/firmware-stuff/emba_logs_linksys_whw03_jnap/l25_web_checks/crawling_192.168.0.1-52000-200ok.log

After:
56 /home/m1k3/firmware-stuff/emba_logs_linksys_whw03/l25_web_checks/crawling_192.168.1.1-52000-200ok.log

@BenediktMKuehne BenediktMKuehne merged commit 1c785ce into e-m-b-a:master Sep 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BenediktMKuehne BenediktMKuehne BenediktMKuehne approved these changes

Assignees

@m-1-k-3 m-1-k-3

Labels
EMBA enhancement New feature or request System emulator (Lxx) The system emulator (Lxx modules)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@m-1-k-3 @BenediktMKuehne