Develop by Benevanio · Pull Request #26 · Cla-Code-Community/candidate

Benevanio · 2026-03-24T22:59:28Z

No description provided.

HUDSON ADDITION

Copilot

Pull request overview

This PR expands the job-scraping pipeline to support multiple sources (via adapters), adds an in-memory cache + request de-duplication layer, and exposes a new API endpoint to search jobs with caching; it also updates exports/UI to include the job “source”.

Changes:

Introduces a multi-source scraping pipeline (sources/, pipeline/) with caching (cache/) and request de-duplication.
Adds /api/jobs/search endpoint returning cached search results and updates CLI run flow to use the new pipeline.
Updates exports/UI/types to include a source field and adds extensive unit test coverage for the new components.

Reviewed changes

Copilot reviewed 37 out of 39 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
frontend/src/types/jobs.ts	Adds `source` field to the frontend `Job` type.
frontend/src/components/JobsTableCard.tsx	Displays the new “Fonte” column in the jobs table.
frontend/package.json	Adds `rolldown` dependency and reorders some devDependencies.
docker-compose.yml	Changes build contexts/dockerfiles and backend start command.
cspell.json	Adds a workspace cspell config file.
backend/tests/unit/utils/config.test.js	Updates expected default output filenames.
backend/tests/unit/services/pipeline/searchJobsWithCache.test.js	Adds unit tests for caching behavior and cache keys.
backend/tests/unit/services/pipeline/scrapeAllSources.test.js	Adds unit tests for normalization, dedupe, logging, and error handling across adapters.
backend/tests/unit/services/pipeline/requestDedup.test.js	Adds unit tests for request de-duplication behavior.
backend/tests/unit/services/linkedinScraper.test.js	Removes legacy LinkedIn scraper tests (old module).
backend/tests/unit/services/cache/cache.test.js	Adds unit tests for `MemoryCache` TTL/CRUD behavior.
backend/tests/unit/services/app.test.js	Updates app test to mock the new pipeline + sources.
backend/tests/unit/services/adapters/theMuse.test.js	Adds unit tests for The Muse adapter behavior and URL construction.
backend/tests/unit/services/adapters/linkedin.test.js	Adds unit tests for LinkedIn adapter parsing and error handling.
backend/tests/unit/services/adapters/lever.test.js	Adds unit tests for Lever adapter filtering/normalization.
backend/tests/unit/services/adapters/greenhouse.test.js	Adds unit tests for Greenhouse adapter filtering/normalization.
backend/tests/unit/services/adapters/adzuna.test.js	Adds unit tests for Adzuna adapter behavior and URL construction.
backend/src/sources/index.js	Defines the enabled sources list (LinkedIn, Adzuna, The Muse, ATS builders).
backend/src/sources/green_lever_builder.js	Builds Greenhouse/Lever adapters from company lists.
backend/src/pipeline/searchJobsWithCache.js	Implements cache-key building + caching around `scrapeAllSources`.
backend/src/pipeline/scrapeAllSources.js	Scrapes all adapters/keywords, normalizes jobs, and deduplicates results.
backend/src/pipeline/requestDedup.js	Adds in-process in-flight promise sharing by cache key.
backend/src/jobsApiApp.js	Adds `/api/jobs/search` endpoint using the new pipeline + sources.
backend/src/interface/lever.interface.js	Adds curated Lever company slugs list.
backend/src/interface/index.js	Exposes greenhouse/lever company lists.
backend/src/interface/greenhouse.interface.js	Adds curated Greenhouse company board tokens list.
backend/src/exporter.js	Updates PDF export header and adds `source` column to output.
backend/src/config.js	Changes default output filenames to `vagas_remoto.*`.
backend/src/cache/cache.js	Adds `MemoryCache` implementation and exports a singleton cache.
backend/src/app.js	Updates CLI run flow to use `searchJobsWithCache(sources, config)`.
backend/src/adapters/theMuse.js	Adds The Muse adapter.
backend/src/adapters/linkedin.js	Refactors LinkedIn scraping into an adapter with `search()`.
backend/src/adapters/lever.js	Adds Lever adapter factory for company slugs.
backend/src/adapters/greenhouse.js	Adds Greenhouse adapter factory for board tokens.
backend/src/adapters/adzuna.js	Adds Adzuna adapter factory and URL builder.
backend/.env.example	Updates example output filenames to `vagas_remoto.*`.
.cspell/custom-dictionary-workspace.txt	Adds/updates custom dictionary words for spellchecking.

Comments suppressed due to low confidence (1)

backend/src/adapters/linkedin.js:129

logWarn only accepts a single message argument (see src/logger.js), so passing a second argument here drops the error details. If you want to log the underlying error, interpolate it into the message string (or extend the logger API).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-24T23:04:28Z

    build:
-      context: ./frontend
-      dockerfile: Dockerfile
+      context: .
+      dockerfile: frontend/Dockerfile
    container_name: vagas-frontend


Same issue for the frontend service: build.context: . combined with frontend/Dockerfile (COPY package*.json ./) will pull the root package*.json instead of frontend/package*.json, likely breaking npm ci / caching. Consider reverting context to ./frontend or updating the Dockerfile to copy/install from the frontend/ workspace.

Copilot · 2026-03-24T23:04:29Z

+            setTimeout(resolve, config.waitBetweenSearchesMs || 1000),
+          );
+        } catch (error) {
+          console.error("Adzuna URL:", url);


The catch block logs the full Adzuna URL, which includes app_id and app_key query params. This can leak credentials into logs. Please remove the full URL from logs or redact sensitive query parameters before logging (and consider using the shared logger instead of console.error).

Suggested change

console.error("Adzuna URL:", url);

const redactedUrl = new URL(url);

redactedUrl.searchParams.delete("app_id");

redactedUrl.searchParams.delete("app_key");

console.error("Adzuna URL:", redactedUrl.toString());

Copilot · 2026-03-24T23:04:29Z

+      { key: "local", label: "Local", width: 100 },
+      { key: "link", label: "Link", width: 280 },
    ];



The PDF column widths now sum to ~860pt, which exceeds A4 landscape usable width (~842-2*30 ≈ 782pt). This will overflow/clamp content off-page. Please rebalance/reduce widths (likely link/titulo) or compute widths dynamically based on page width.

Suggested change

// Ensure total column width does not exceed the usable page width.

const usableWidth =

doc.page.width - doc.page.margins.left - doc.page.margins.right;

const totalWidth = cols.reduce((sum, col) => sum + col.width, 0);

if (totalWidth > usableWidth && totalWidth > 0) {

const scale = usableWidth / totalWidth;

cols.forEach((col) => {

col.width = Math.floor(col.width * scale);

});

}

Copilot · 2026-03-24T23:04:29Z

+      const config = {
+        ...getConfig(),
+        keywords: req.query.keywords
+          ? String(req.query.keywords)
+              .split(",")
+              .map((k) => k.trim())
+          : getConfig().keywords,


This endpoint calls getConfig() twice and also allows keywords= to produce [''] (no .filter(Boolean)), which can trigger searches with an empty keyword. Reuse the first config object and normalize/sanitize the query keywords (trim + filter empty + optional de-dupe).

Suggested change

const config = {

...getConfig(),

keywords: req.query.keywords

? String(req.query.keywords)

.split(",")

.map((k) => k.trim())

: getConfig().keywords,

const baseConfig = getConfig();

let keywords;

if (req.query.keywords !== undefined) {

const rawKeywords = String(req.query.keywords)

.split(",")

.map((k) => k.trim())

.filter(Boolean);

keywords =

rawKeywords.length > 0 ? Array.from(new Set(rawKeywords)) : baseConfig.keywords;

} else {

keywords = baseConfig.keywords;

}

const config = {

...baseConfig,

keywords,

Copilot · 2026-03-24T23:04:30Z

+  for (const adapter of adapters) {
+    logInfo(`Iniciando fonte: ${adapter.sourceName}`);
+


logInfo uses adapter.sourceName directly; adapters without sourceName will log "Iniciando fonte: undefined" (and this file itself allows sourceName to be missing in tests). Consider using a fallback like adapter.sourceName ?? 'unknown' to keep logs consistent.

Copilot · 2026-03-24T23:04:30Z

    build:
-      context: ./backend
-      dockerfile: Dockerfile
+      context: .
+      dockerfile: backend/Dockerfile
    container_name: vagas-backend
-    command: ["node", "src/server.js"]
+    command: ["node", "backend/index.js"]
    env_file:


Backend container command now runs node backend/index.js, which executes the one-off scraper (src/app.js) and does not start the API server (src/server.js). This will break the frontend’s VITE_API_PROXY_TARGET=http://backend:3001 and the exposed port. Also, using build.context: . with backend/Dockerfile (COPY package*.json ./) will copy the root package files, likely breaking npm ci / workspace installs. Consider reverting to context: ./backend + command: ["node", "src/server.js"], or update the Dockerfile/command consistently to run the server from the root context (e.g. backend/src/server.js).

Copilot · 2026-03-24T23:04:31Z

+    drawRow(
+      {
+        palavra: "Palavra-chave",
+        titulo: "Título",
+        empresa: "Empresa",
+        local: "Local",
+        link: "Link",
+      },
+      true,
+    );


The first-page header row passed to drawRow does not include the new source column label, so the "Fonte" header cell will render blank on the first page (it’s only present on subsequent pages). Include source: "Fonte" in the initial header object as well.

Copilot · 2026-03-24T23:04:31Z

+  return [...keywords]
+    .map((k) => String(k).trim().toLowerCase())
+    .filter(Boolean)
+    .sort();


normalizeKeywords() sorts but does not deduplicate. This means logically equivalent keyword sets (e.g. duplicated entries) generate different cache keys, reducing cache hit rate and potentially triggering redundant scraping. Consider de-duplicating keywords (e.g. via a Set) when building the cache key.

Suggested change

return [...keywords]

.map((k) => String(k).trim().toLowerCase())

.filter(Boolean)

.sort();

return [...new Set(

[...keywords]

.map((k) => String(k).trim().toLowerCase())

.filter(Boolean),

)].sort();

Copilot · 2026-03-24T23:04:31Z

+export const sources = [
+  linkedinAdapter,
+
+  createAdzunaAdapter({
+    appId: process.env.ADZUNA_APP_ID || "",
+    appKey: process.env.ADZUNA_APP_KEY || "",
+    country: process.env.ADZUNA_COUNTRY || "br",
+  }),


createAdzunaAdapter is always included even when ADZUNA_APP_ID / ADZUNA_APP_KEY are empty strings. That will cause repeated failing HTTP calls and noisy warnings for every run. Consider only adding the Adzuna source when both credentials are provided (or make the adapter return [] early when not configured).

Suggested change

export const sources = [

linkedinAdapter,

createAdzunaAdapter({

appId: process.env.ADZUNA_APP_ID || "",

appKey: process.env.ADZUNA_APP_KEY || "",

country: process.env.ADZUNA_COUNTRY || "br",

}),

const adzunaAppId = process.env.ADZUNA_APP_ID;

const adzunaAppKey = process.env.ADZUNA_APP_KEY;

export const sources = [

linkedinAdapter,

adzunaAppId &&

adzunaAppKey &&

createAdzunaAdapter({

appId: adzunaAppId,

appKey: adzunaAppKey,

country: process.env.ADZUNA_COUNTRY || "br",

}),

Copilot · 2026-03-24T23:04:31Z

    "react": "^19.2.4",
    "react-dom": "^19.2.4",
+    "rolldown": "^1.0.0-rc.10",
    "tailwind-merge": "^3.5.0"


rolldown is added as a production dependency but there are no references to it in the frontend source/scripts. If it isn’t used, it should be removed to avoid unnecessary install size/optional native bindings; if it is a build-time tool, it likely belongs in devDependencies instead.

hltav and others added 7 commits March 24, 2026 16:20

Adding new data sources for searches.

c279ca9

Correction the tests for the new structure

f852863

Correcting the tests for the new structure.

2f0548c

Correcting the tests for the new structure.

e00c6a8

Adding new branch tests.

5206f90

merge: sync feature branch with hudson/master

d9281dd

Feature/hudson (#25)

fd6c50c

HUDSON ADDITION

Copilot AI review requested due to automatic review settings March 24, 2026 22:59

Copilot started reviewing on behalf of Benevanio March 24, 2026 22:59 View session

Copilot AI reviewed Mar 24, 2026

View reviewed changes

Benevanio merged commit f6fdee6 into master Mar 24, 2026
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Develop#26

Develop#26
Benevanio merged 7 commits into
masterfrom
develop

Benevanio commented Mar 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-          console.error("Adzuna URL:", url);
+          const redactedUrl = new URL(url);
+          redactedUrl.searchParams.delete("app_id");
+          redactedUrl.searchParams.delete("app_key");
+          console.error("Adzuna URL:", redactedUrl.toString());

+    // Ensure total column width does not exceed the usable page width.
+    const usableWidth =
+      doc.page.width - doc.page.margins.left - doc.page.margins.right;
+    const totalWidth = cols.reduce((sum, col) => sum + col.width, 0);
+    if (totalWidth > usableWidth && totalWidth > 0) {
+      const scale = usableWidth / totalWidth;
+      cols.forEach((col) => {
+        col.width = Math.floor(col.width * scale);
+      });
+    }

		for (const adapter of adapters) {
		logInfo(`Iniciando fonte: ${adapter.sourceName}`);

Uh oh!

Conversation

Benevanio commented Mar 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants