Target Determinator Indexer Workflow

[osalpekar]

As described in this talk and this repo, we are experimenting with using CodeLlama-powered information retrieval for target determination.

The idea is that we create embeddings for PyTorch test functions, and store this index in S3. Then when a new PR comes in, we create embedding(s) for that PR, compare them to the index of test embeddings, and run only the most relevant tests.

This PR creates a workflow that does the indexing part (creating embeddings for functions and store in S3). All the logic for running the indexer is in osalpekar/llm-target-determinator. This workflow just checks out the relevant repos, installs the dependencies, runs the torchrun command to trigger indexing, and uploads the artifacts to S3.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/118824

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 781075a with merge base 55483fc ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[huydhn]

Just a note, it makes sense to add a timestamp here in indexer zip file, we could also setup a retention rule of N days on S3 to clean up old files. This could be done later though

[osalpekar]

Good point!

[huydhn]

I think you will need to call setup-linux https://github.com/pytorch/pytorch/blob/main/.github/workflows/_linux-test.yml#L71-L72 and install NVIDIA driver https://github.com/pytorch/pytorch/blob/main/.github/workflows/_linux-test.yml#L94-L97 here. If this job is run on a newly launch g5 runner by chance, it would not have CUDA driver IIRC, so the indexing step would fail

[osalpekar]

Hmm that's strange, looks like indexing passed in this job on GPU: https://github.com/pytorch/pytorch/actions/runs/7849903772/job/21424841173?pr=118824. Will investigate in a follow-up.

[huydhn]

Oh, the runner is reusable, as long as the indexing job is not the first one to run, the driver will be there. However, my understanding might be outdated, may be we already have included CUDA driver in the image (although I doubt that)

[huydhn]

Overall LGTM! There is a small bug w.r.t NVIDIA driver installation, but that's an easy fix.

[osalpekar]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here