Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make testing extractors easier #102

Open
diptanu opened this issue Aug 22, 2023 · 0 comments
Open

Make testing extractors easier #102

diptanu opened this issue Aug 22, 2023 · 0 comments

Comments

@diptanu
Copy link
Collaborator

diptanu commented Aug 22, 2023

New Extractors can be tested by running the server and by binding them to a repository. This is cumbersome because the feedback cycle is not fast.

Solution -
Create a sub-command under the indexify binary to run just an extractor

indexify executor run-extractor --python-module foobar.ExtractorClassName --content text-from-which-we-are-extracting --params `{"foo": "bar"}`

The above example allows a developer who is developing a new extractor in footer.ExtractorClassName to load the extractor on their terminal pass some text to it and also the input parameters of the extractors encoded in json.

In the future when we add support for images, videos, or any other blobs, we could enhance this to pass the pointer to files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant