Skip to content

Refactor/vertex ai for scraper#82

Merged
lcai000 merged 11 commits intomainfrom
refactor/vertex-ai-for-scraper
Apr 12, 2026
Merged

Refactor/vertex ai for scraper#82
lcai000 merged 11 commits intomainfrom
refactor/vertex-ai-for-scraper

Conversation

@lcai000
Copy link
Copy Markdown
Collaborator

@lcai000 lcai000 commented Apr 9, 2026

scraper refactor

reason: scraper is chopped and summaries don't work

changes

  • use Google Vertex AI framework for AI summaries (avoid being blocked, google somehow blocks the requests from render...anyways this method is also more industry and scalable)
  • more lenient article handler - before the way some articles were written makes the code block them (format issue)
  • reduce unicode in logs (when image generates, a bunch of unicode is shown)---this may or may not be working

things still need to be done

  • switch to nanobanana for image generation
  • improve summarization schema

@lcai000 lcai000 requested a review from ThatXliner April 9, 2026 03:00
@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 9, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
billion-nextjs Error Error Apr 12, 2026 3:21am

@lcai000
Copy link
Copy Markdown
Collaborator Author

lcai000 commented Apr 9, 2026

Also need to fix issue with Orders

Copy link
Copy Markdown
Collaborator

@ThatXliner ThatXliner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

description field doesn't appear to be used?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wtf? why a whole separate file jsut for this?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

three different files use it for import managing (text-generation.ts, image-keywords.ts, marketing-generation.ts)
and for separation of concerns

trimmed.length > 2)
);
// Blank lines
if (trimmed === "") return true;
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's this for

@lcai000 lcai000 merged commit fdabd3b into main Apr 12, 2026
1 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants