Skip to content

Conversation

matyascimbulka
Copy link
Collaborator

@matyascimbulka matyascimbulka commented Mar 26, 2025

Based on this comment by @metalwarrior665 I've decided to have a look at simplifying input handling.

To simplify things I have decided to remove the defaults object and rewrite the validateAndFillInput function to use the input schema directly.

How it works

Normal mode

In normal mode the input is received via Actor.getInput() which is then processed using the processInput function where its validated and completed. This function then returns the necessary crawler options to complete the query.

Standby mode

When standby mode is initiated it will receive its initial input via Actor.getInput(). This input is processed using the processStandbyInput which returns crawler options for all possible crawlers which is then used to start them. Then it waits for request.
When a request arrives its parameters are parsed using the parseParameters function which returns Partial<Input> object. This object is then directly fed into the parseInput function where its validated and completed like standby input.


Here are example runs:

Copy link
Collaborator

@jirispilka jirispilka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!
Looks great.

As Lukas suggested, can you also remove table with query params from README.md?
Since the inputs for normal mode and standby are 1:1. It will greatly shorten the README.

Otherwise, LGTM.

src/utils.ts Outdated
type SchemaKey = keyof typeof inputSchema.properties;

const typedKey = key as SchemaKey;
// const inputKey = key as keyof typeof parsedInput;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove this commented out code

@matyascimbulka
Copy link
Collaborator Author

@jirispilka I have removed the dead code and remove the table from README.md.

@matyascimbulka matyascimbulka requested review from MQ37 and removed request for metalwarrior665 March 31, 2025 12:15
Copy link
Contributor

@MQ37 MQ37 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@matyascimbulka
Copy link
Collaborator Author

Thanks I'll merge and deploy the changes.

@MQ37 This will probably break your other PR (#69)

@matyascimbulka matyascimbulka merged commit 358c6d7 into master Mar 31, 2025
1 check passed
@matyascimbulka matyascimbulka deleted the refactor/simplify-input-handling branch March 31, 2025 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants