Skip to content

Improve web page crawler - #795

Merged
BenediktMKuehne merged 1 commit into
e-m-b-a:masterfrom
m-1-k-3:http_crawler
Sep 19, 2023
Merged

Improve web page crawler#795
BenediktMKuehne merged 1 commit into
e-m-b-a:masterfrom
m-1-k-3:http_crawler

Conversation

@m-1-k-3

@m-1-k-3 m-1-k-3 commented Sep 19, 2023

Copy link
Copy Markdown
Member
  • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

Feature

  • What is the current behavior? (You can also link to an open issue here)

On some web servers we get 200 ok for pages that are not available

  • What is the new behavior (if this is a feature change)? If possible add a screenshot.

We do a dummy random request to identify a not available response and use this to compare to the further responses.

@m-1-k-3 m-1-k-3 added enhancement New feature or request System emulator (Lxx) The system emulator (Lxx modules) EMBA labels Sep 19, 2023
@m-1-k-3 m-1-k-3 self-assigned this Sep 19, 2023
@m-1-k-3

m-1-k-3 commented Sep 19, 2023

Copy link
Copy Markdown
Member Author

On my testfirmware we improved the results massively:

Before:
16726 /home/m1k3/firmware-stuff/emba_logs_linksys_whw03_jnap/l25_web_checks/crawling_192.168.0.1-52000-200ok.log

After:
56 /home/m1k3/firmware-stuff/emba_logs_linksys_whw03/l25_web_checks/crawling_192.168.1.1-52000-200ok.log

@BenediktMKuehne
BenediktMKuehne merged commit 1c785ce into e-m-b-a:master Sep 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BenediktMKuehne BenediktMKuehne BenediktMKuehne approved these changes

Assignees

@m-1-k-3 m-1-k-3

Labels

EMBA enhancement New feature or request System emulator (Lxx) The system emulator (Lxx modules)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@m-1-k-3 @BenediktMKuehne