Annotations: Ignore unique constraint violations for tags

[sakjur]

What is this feature?

When creating annotations with tags very fast, there's a race condition where the tag table might have tags added simultaneously triggering a unique constraint violation.

Considerations:

• Explicit database locks increased complexity quite a bit for relatively little gain.
• A regular mutex would probably work really well in practice, but HA instances could irregularly still experience the problem.

Instead, I opted for a single retry which seems to work well according to my testing using k6 (I'm not able to write a reliable reproduction test in Go, since this depends on a race condition).

Who is this feature for?

Annotations API users.

Which issue(s) does this PR fix?:

Fixes #61203

Special notes for your reviewer:

excerpt from k6 script for duplication and validation

export default function (data) {
    login("admin", "admin");

    const ts = Math.floor(Date.now() / 100)
    const res = http.post(http.url`${url}/api/annotations`,
        JSON.stringify({
            dashboardUID: data.dashboards[0],
            time: Date.now(),
            timeEnd: Date.now()+30000,
            tags: ['project=foo', `identifier=${ts}`],
            text: 'Wrote an annotation',
        }),
        {
            tags: { type: "annotations", op: "create" },
            headers: { "Content-Type": "application/json" },
        }
    );

    check(res, {
        "annotation successfully created": (r) => r.status === 200,
        "creating annotation takes less than 150ms": (r) => r.timings.duration < 150,
    });

    if (res.status >= 500) {
        console.log(res)
    }
}

Please check that:

• It works as expected from a user's perspective.
• If this is a pre-GA feature, it is behind a feature toggle.
• The docs are updated, and if this is a notable improvement, it's added to our What's New doc.