Skip to content
This repository was archived by the owner on Jun 5, 2025. It is now read-only.

Add malicious package test for vllm #912

Merged
merged 1 commit into from
Feb 5, 2025
Merged

Add malicious package test for vllm #912

merged 1 commit into from
Feb 5, 2025

Conversation

rdimitrov
Copy link
Member

@rdimitrov rdimitrov commented Feb 4, 2025

The following PR adds malicious package test for vllm.

Related to: #904

aponcedeleonch
aponcedeleonch previously approved these changes Feb 5, 2025
@rdimitrov rdimitrov dismissed aponcedeleonch’s stale review February 5, 2025 08:20

hey, let's not merge it yet, it doesn't seem stable enough

Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com>
@rdimitrov
Copy link
Member Author

I had to update the malicious package test case for this and the rest of the local providers because of the capabilities of the 0.5B model we are using for these tests. It looks like it cannot handle the system prompts as precisely as we expect which results in non-deterministic responses, i.e. sometimes it includes example code snippets even though it says the package is malicious, sometimes it doesn't include the link to insights. To handle all of this decided to simplify the test and just check if CodeGate was able to flag it.

Copy link
Contributor

@jhrozek jhrozek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the functionality is shared across the providers, so I think this is good enough

@rdimitrov rdimitrov merged commit 615a838 into main Feb 5, 2025
9 checks passed
@rdimitrov rdimitrov deleted the malicious-vllm branch February 5, 2025 12:17
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants