Dev by amrelsagaei · Pull Request #1 · caido-community/crawler

amrelsagaei · 2026-01-13T12:42:23Z

No description provided.

socket-security · 2026-01-13T12:43:01Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@caido/tailwindcss@0.0.1
	vue-tsc@3.1.1
	@caido/eslint-config@0.6.0
	@vitest/coverage-v8@4.0.17
	@caido-community/dev@0.1.6
	@caido/sdk-backend@0.55.0-beta.0
	@caido/primevue@0.3.3
	vitest@4.0.17
	@caido/sdk-frontend@0.54.2-beta.10
	pinia@3.0.4
	postcss-prefixwrap@1.57.0
	tailwindcss-primeui@0.3.4
	@codemirror/view@6.38.5
	tailwindcss@3.4.13
	typescript@5.5.4
	@vitejs/plugin-vue@5.2.1
	vue@3.5.22
	knip@5.70.2
	primevue@4.1.0
	eslint@9.29.0
	zod@4.3.5

View full report

socket-security · 2026-01-13T12:43:03Z

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action	Severity	Alert (click "▶" to expand/collapse)
Warn		Obfuscated code: npm `entities` is 91.0% likely obfuscated Confidence: 0.91 Location: Package overview From: pnpm-lock.yaml → `npm/vue@3.5.22` → `npm/entities@4.5.0` ℹ Read more on: This package \| This alert \| What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at `support@socket.dev`. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment `@SocketSecurity ignore npm/entities@4.5.0`. You can also ignore all packages with `@SocketSecurity ignore-all`. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Obfuscated code: npm `safer-buffer` is 94.0% likely obfuscated Confidence: 0.94 Location: Package overview From: pnpm-lock.yaml → `npm/@caido-community/dev@0.1.6` → `npm/safer-buffer@2.1.2` ℹ Read more on: This package \| This alert \| What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at `support@socket.dev`. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment `@SocketSecurity ignore npm/safer-buffer@2.1.2`. You can also ignore all packages with `@SocketSecurity ignore-all`. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Obfuscated code: npm `vite` is 91.0% likely obfuscated Confidence: 0.91 Location: Package overview From: pnpm-lock.yaml → `npm/vitest@4.0.17` → `npm/@vitejs/plugin-vue@5.2.1` → `npm/@caido-community/dev@0.1.6` → `npm/vite@6.0.7` ℹ Read more on: This package \| This alert \| What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at `support@socket.dev`. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment `@SocketSecurity ignore npm/vite@6.0.7`. You can also ignore all packages with `@SocketSecurity ignore-all`. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

amrelsagaei · 2026-01-14T15:07:18Z

here i have implemented:

Project Setup & Configuration
workspace dependencies between packages
Shared Types & Schemas
Backend Configuration Store
Crawl Engine Core
Crawl Job Management

Corb3nik

There's a lot to review here, so I'll start with the big structural changes.

May I suggest to use the same structure as we do for proxy-frontend and other projects?

In the backend plugin:

api/
services/
stores/
- job store
- project store
- config store
- [move any global variables into stores]
repositories/
- HTTPQClient code
models/ (contains base objects like Request, ...)
parsers/ (this is basically your extractors folder)

Corb3nik

Ok this was a tough PR to review, mostly because the backend code is very messy.

I added some comments, but there's still a lot more work to be done. Maybe we can merge/release this even if it's not perfect.

One more thing:

After fixing the comments, could you add tests for each model and extractor? There's a lot of complicated code in there.

Corb3nik · 2026-01-17T12:54:03Z

+      jobsStore.updateJob(jobId, {
+        crawledUrls: stats.crawledUrls,
+        discoveredUrls: stats.discoveredUrls,
+      });


Let's move the jobsStore call to the service.

Corb3nik · 2026-01-17T12:54:11Z

+      });
+    },
+    onComplete: (stats, jobId) => {
+      jobsStore.updateJob(jobId, {


Corb3nik · 2026-01-19T11:43:17Z

+    this.recentSuccessRate =
+      this.completedTasks / (this.completedTasks + this.failedTasks);
+  }


I think we should remove recentSuccessRate since we can calculate it later when we need it (like in the frontend)

Corb3nik · 2026-01-19T11:44:25Z

+    this.totalTaskDuration += duration;
+    this.recentDurations.push(duration);
+
+    if (this.recentDurations.length > 100) {
+      this.recentDurations.shift();
+    }


Do we need to track totalTaskDuration?

Also do we need to set a limit on the recent durations?

What we could do is just track all of the durations, and calculate the totalTaskDuration later in the frontend when we need it.

Corb3nik · 2026-01-19T11:44:45Z

+    this.totalTaskDuration += duration;
+    this.recentDurations.push(duration);
+
+    if (this.recentDurations.length > 100) {
+      this.recentDurations.shift();
+    }


Same comment.

Corb3nik · 2026-01-19T12:25:22Z

+
+  private buildUrl(
+    request: Parameters<
+      Parameters<BackendSDK["events"]["onInterceptResponse"]>[0]


Corb3nik · 2026-01-19T12:32:04Z

+  private options: Required<
+    Omit<
+      CrawlerOptions,
+      | "requestQueue"
+      | "router"
+      | "requestHandler"
+      | "preNavigationHooks"
+      | "postNavigationHooks"
+      | "failedRequestHandler"
+      | "sessionPoolOptions"
+    >
+  > & {
+    requestQueue: RequestQueue;
+    router: Router;
+    requestHandler?: (context: CrawlingContext) => Promise<void>;
+    preNavigationHooks: PreNavigationHook[];
+    postNavigationHooks: PostNavigationHook[];
+    failedRequestHandler?: FailedRequestHandler;
+    sessionPoolOptions: CrawlerOptions["sessionPoolOptions"];
+  };


This should be its own HTTPCrawlerOptions type. This code is very difficult to read/understand/maintain.

Corb3nik · 2026-01-19T12:35:22Z

+ * CrawlerStore - Manages active crawler instances
+ */
+
+import type { HttpCrawler } from "../services/crawler";


If the HttpCrawler class can be stored in a store, that means that it should we move it to the models folder.

The architecture we're going for here is:

API imports the services

Service imports the store/repository

API/Services/Stores/Repositories import from the models

The stores should never import from the services like we're doing here.

Corb3nik · 2026-01-19T12:43:14Z

+  plugins: [
+    {
+      kind: "backend",
+      id: "backend",


Lets' update the ID to crawler-backend

Corb3nik · 2026-01-19T12:43:19Z

+    },
+    {
+      kind: 'frontend',
+      id: "frontend",


crawler-frontend

amrelsagaei · 2026-01-19T15:00:23Z

Please Do Not Review YET!

Corb3nik

Gj, this looks great.

Having the tests will help us a lot to make sure the code doesn't have bugs.

I added a few nits to fix, but I think we're good to go after that.

init

a8d1e40

.

281738a

amrelsagaei changed the title ~~init~~ Dev Jan 14, 2026

amrelsagaei requested review from Corb3nik and Sytten January 14, 2026 15:07

Corb3nik requested changes Jan 15, 2026

View reviewed changes

Comment thread packages/backend/src/engine/crawlers/httpCrawler.ts Outdated

Corb3nik requested changes Jan 15, 2026

View reviewed changes

Comment thread packages/backend/src/api/jobs.ts

Fix The Reviewer Suggestions

06db2ed

amrelsagaei requested a review from Corb3nik January 17, 2026 00:20

amrelsagaei added 4 commits January 17, 2026 07:23

Crawler Settings

33dd390

Auto-Crawl Listener

8380e2f

V1

cc37d08

Github Actions

c4877d4

Corb3nik requested changes Jan 19, 2026

View reviewed changes

amrelsagaei added 2 commits January 19, 2026 11:56

resolving comments

17cdc28

typecheck

0ea6bb0

Spliting Crawler.ts and adding tests

d3e6442

amrelsagaei requested a review from Corb3nik January 22, 2026 08:53

Corb3nik requested changes Jan 22, 2026

View reviewed changes

Resolve types

5c6698d

Corb3nik requested review from Corb3nik and removed request for Sytten January 24, 2026 12:39

Corb3nik approved these changes Jan 24, 2026

View reviewed changes

amrelsagaei merged commit 834551e into main Jan 24, 2026
5 checks passed

amrelsagaei deleted the dev branch January 24, 2026 13:06

amrelsagaei restored the dev branch February 7, 2026 12:17

Conversation

amrelsagaei commented Jan 13, 2026

Uh oh!

socket-security Bot commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

socket-security Bot commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amrelsagaei commented Jan 14, 2026

Uh oh!

Corb3nik left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Corb3nik left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amrelsagaei commented Jan 19, 2026

Uh oh!

Corb3nik left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

socket-security Bot commented Jan 13, 2026 •

edited

Loading

socket-security Bot commented Jan 13, 2026 •

edited

Loading

Corb3nik left a comment •

edited

Loading

Corb3nik left a comment •

edited

Loading