chore: only remove labels during stable releases

[mcansh]

No description provided.

[changeset-bot]

⚠️ No Changeset found

Latest commit: bafef9a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions]

🤖 Hello there,

We just published version v0.4.1 which includes this pull request. If you'd like to take it for a test run please try it out and let us know what you think!

Thanks!