Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to event-driven infrastructure and improve support for asynchronous questions #632

Merged
merged 170 commits into from Apr 11, 2024

Conversation

cortadocodes
Copy link
Member

@cortadocodes cortadocodes commented Mar 5, 2024

Summary

This pull request:

  • Makes the SDK fully event-driven by using a single topic to emit/consume events
  • Majorly refactors the event handler to facilitate asynchronous event retrieval
  • Adds the ability to get and replay events from a BigQuery store

Contents (#632)

IMPORTANT: There are 4 breaking changes.

New features

  • 💥 BREAKING CHANGE: Use single topic per workspace #639
  • Add get_events function for retrieving events asynchronously from BigQuery
  • Add EventReplayer class to replay asynchronously-retrieved events
  • Add Manifest.download method

Enhancements

  • Get subscription project name from topic by default
  • Improve asking of asynchronous questions via Child.ask
  • Return download path from Dataset.download
  • Include question UUID in delivery acknowledgement log message
  • Improve handling of invalid events
  • Add datetime and uuid attributes to all events

Fixes

  • Await successful publishing of question messages
  • Fix api_access_endpoint usage in mock_generate_signed_url

Operations

  • Add test BigQuery dataset, cloud function, and IAM roles to terraform config
  • Switch to reusable workflows where possible

Dependencies

  • Add google-cloud-bigquery
  • Upgrade coolname
  • Add db-dtypes for converting bigquery rows to dataframes

Refactoring

  • 💥 BREAKING CHANGE: Rename x.received_messages to x.received_events
  • 💥 BREAKING CHANGE: Rename record_messages parameters to record_events
  • 💥 BREAKING CHANGE: Update ChildEmulator to use event* instead of message*
  • Factor out making minimal dictionary
  • Factor out creating push subscription
  • Factor out emitting question event in Service.ask
  • Factor out event handlers and related logic from OrderedMessageHandler into new AbstractEventHandler
  • Move validation module into octue.cloud.events subpackage
  • Rename OrderedMessageHandler to GoogleCloudPubSubEventHandler
  • Rename "message" to "event" in event handler classes
  • Rename GooglePubSubHandler to GoogleCloudPubSubHandler

Chores

  • Update licence year to 2024

Testing

  • Simplify various tests

Upgrade instructions

  • Update all services in your services network to this version of octue or later (0.53.0+)
  • Replace any usages of the received_messages methods with received_events
  • Replace any usages of the record_messages parameters with record_events
  • Replace the word message with event in usages of ChildEmulator methods (apart from in the case of monitor_message)

@cortadocodes cortadocodes self-assigned this Mar 5, 2024
@cortadocodes cortadocodes linked an issue Mar 5, 2024 that may be closed by this pull request
25 tasks
@cortadocodes cortadocodes force-pushed the better-support-asynchronous-questions branch from 708b975 to fe2dafb Compare March 5, 2024 18:59
@cortadocodes cortadocodes changed the title Better support asynchronous questions Improve support for asynchronous questions Mar 6, 2024
BREAKING CHANGE: Replace the `record_messages` parameter with `record_events` and the `received_messages` property with `received_events`.
BREAKING CHANGE: Upgrade all services in your network to this version of `octue` or above.
BREAKING CHANGE: Replace any usage of the word `message` to `event` in any arguments, method names, and files containing events.
@cortadocodes cortadocodes merged commit 203ed7b into main Apr 11, 2024
5 checks passed
@cortadocodes cortadocodes deleted the better-support-asynchronous-questions branch April 11, 2024 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update licence year to 2024 Improve support for asynchronous questions
2 participants