Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modified CI to allow vector input #76

Merged
merged 3 commits into from
May 23, 2024
Merged

Modified CI to allow vector input #76

merged 3 commits into from
May 23, 2024

Conversation

chun9l
Copy link
Collaborator

@chun9l chun9l commented May 21, 2024

Some benchmark fails in CI because the inputs are unrealistic. Atm users can only specify one value to fill the whole vector. Changes introduce here will allow users to set an input vector; the original behavior is retained.

Specify them like this --input_value="[input1, input2,...]"

@chun9l chun9l marked this pull request as draft May 21, 2024 17:18
@chun9l chun9l marked this pull request as ready for review May 21, 2024 17:29
@linusseelinger
Copy link
Member

Looks good! Does this also work with models that have multiple input vectors?

@chun9l
Copy link
Collaborator Author

chun9l commented May 22, 2024

Ah good point, I didn't thought of that. Let me see if there's an existing model with this type of input

@chun9l
Copy link
Collaborator Author

chun9l commented May 22, 2024

The answer to your question was no sadly:(

But this new push should do the trick. I ditched json.loads and used eval instead. Also, altered the for loop in test_evaluate to account for nested lists.

For some reason though, spaces are not allowed when specifying input_value. May be due to how Python/Docker parses the string

@annereinarz annereinarz merged commit fec642b into main May 23, 2024
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants