Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move Feedback Ingestion to its own Topic / Consumer #66100

Closed
4 tasks done
Tracked by #64671
JoshFerge opened this issue Feb 29, 2024 · 5 comments
Closed
4 tasks done
Tracked by #64671

Move Feedback Ingestion to its own Topic / Consumer #66100

JoshFerge opened this issue Feb 29, 2024 · 5 comments
Assignees
@aliu3ntry

Comments

@JoshFerge
Copy link
Member

JoshFerge commented Feb 29, 2024 •
edited by aliu3ntry

Tasks
Beta Give feedback

Motivation: when errors pipeline has problems, we don't want to lose feedbacks. Additionally, we can add things like a dead letter queue, etc.
@JoshFerge JoshFerge mentioned this issue Feb 29, 2024
Open
@cmanallen
Copy link
Member

@aliu-at-sentry You've been volunteered for this issue but don't worry! I'll pair with you and we can tackle it together. Let's plan to start on this on Monday. We can use our first standup as the jumping off point.

@aliu3ntry aliu3ntry mentioned this issue Mar 1, 2024
Closed
aliu3ntry added a commit to getsentry/sentry-kafka-schemas that referenced this issue Mar 1, 2024
@aliu3ntry
Create ingest-feedbacks.yaml
766ba05 
https://getsentry.atlassian.net/browse/OPS-5317
getsentry/sentry#66100

Pipeline/cluster = replays, since replay team owns user feedback. We expect the volume to be too low to justify its own cluster.
@aliu3ntry aliu3ntry mentioned this issue Mar 1, 2024
Merged
aliu3ntry added a commit to getsentry/sentry-kafka-schemas that referenced this issue Mar 5, 2024
@aliu3ntry
Create ingest-feedbacks.yaml
c500a24 
https://getsentry.atlassian.net/browse/OPS-5317
getsentry/sentry#66100

Pipeline/cluster = replays, since replay team owns user feedback. We expect the volume to be too low to justify its own cluster.
@aliu3ntry aliu3ntry mentioned this issue Mar 6, 2024
Merged
aliu3ntry added a commit to getsentry/sentry-kafka-schemas that referenced this issue Mar 7, 2024
@aliu3ntry
Create ingest-feedbacks.yaml
b2366ed 
https://getsentry.atlassian.net/browse/OPS-5317
getsentry/sentry#66100

Pipeline/cluster = replays, since replay team owns user feedback. We expect the volume to be too low to justify its own cluster.
aliu3ntry added a commit to getsentry/sentry-kafka-schemas that referenced this issue Mar 12, 2024
@aliu3ntry @getsantry
feat(replay): Config new ingest-feedback-events topic (#227)
e74bfe4 
* Create ingest-feedbacks.yaml

https://getsentry.atlassian.net/browse/OPS-5317
getsentry/sentry#66100

Pipeline/cluster = replays, since replay team owns user feedback. We expect the volume to be too low to justify its own cluster.

* Update ingest-feedbacks description

* Fix ingest-feedbacks.yaml producers

+small change to descrip

* add new schema and pipeline + rename

- move to new 'user-feedback' pipeline
- add an initial schema generated from a widget feedback (captured by chrome dev tools) and https://github.com/glideapps/quicktype
- rename to 'ingest-feedback-events' since 'FeedbackEvent' seems to be the corresponding type in Django + SDK/typescript

* style(lint): Auto commit lint changes

* add topic to CODEOWNERS

* change schema ver to draft-07

* remove field titles and add schema title, remove owners-ingest

* Add react ex and make all optional except contexts:{feedback:{contact_email, message, name, source}}

- all fields have additionalProperties = true, making schema totally flexible and backward/forward-compatible
- rename examples folder

* style(lint): Auto commit lint changes

* style(lint): Auto commit lint changes

* modify schema to match sdk event type

* style(lint): Auto commit lint changes

* style(lint): Auto commit lint changes

---------

Co-authored-by: getsantry[bot] <66042841+getsantry[bot]@users.noreply.github.com>
aliu3ntry added a commit that referenced this issue Mar 13, 2024
@aliu3ntry
feat(replay): add config for new topic ingest-feedback-events (#66484)
9ab40c3 
Linked to #66100 and
https://getsentry.atlassian.net/browse/OPS-5317

Dependent on:
[sentry-kafka-schemas
PR](getsentry/sentry-kafka-schemas#227)
@aliu3ntry
Copy link
Member

IMG_7242
Red is TODO

@JoshFerge
Copy link
Member Author

JoshFerge commented Mar 20, 2024 •
edited by aliu3ntry

Relay TODO:

  • Add an option backed feature flag an option in sentry/getsentry, make sure it is available in relay
  • add a KafkaTopic to the kafka topic enum
  • Modify services/store.rs so that it produces ItemType::UserReportV2 to the new kafka topic. sidenote: Attachments should still go to the attachments topic
  • Separate attachments and feedback in the same envelope, send each to their respective topics. See issue above ^
    • Remove option flag once tested
  • Remove option flag once tested

@aliu3ntry aliu3ntry mentioned this issue Mar 27, 2024
Merged
@bruno-garcia bruno-garcia mentioned this issue Mar 27, 2024
Merged
@aliu3ntry aliu3ntry mentioned this issue Mar 27, 2024
Merged
aliu3ntry added a commit that referenced this issue Mar 28, 2024
@aliu3ntry
feat(feedback): add option for ingest feedback topic (#67839)
7e3ded7 
relates to #66100
aliu3ntry added a commit to getsentry/relay that referenced this issue Apr 9, 2024
@aliu3ntry
feat(feedback): produce to ingest-feedback-events topic (#3344)
01311a4 
Option PR:
depends on getsentry/sentry#67839 (MERGED)

Feature flag PRs (outdated/unused):
~getsentry/getsentry#13426 (REVERTED)~
~getsentry/sentry#67747 (REVERTED)~
~getsentry/sentry-options-automator#973
(CLOSED)~

Relates to getsentry/sentry#66100
This was referenced Apr 17, 2024
Closed
Closed
@aliu3ntry
Copy link
Member

aliu3ntry commented Apr 19, 2024 •
edited

Metrics to track ingest in s4s:
https://p.datadoghq.com/sb/FH6-Y3-fdb1541238e21113f4febca249245966

the test project I used

@aliu3ntry
Copy link
Member

🎉🎉🎉🎉🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aliu3ntry aliu3ntry

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@JoshFerge @cmanallen @aliu3ntry