feat: Add support for error notifications for Snowpipe

[gouline]

Adding support for error notifications for Snowpipe.

This feature is currently in private preview and must be enabled for the specific account before these changes can work. Once the feature is released (not sure when), it should work for everybody.

The setup consists of a new outbound type of notification integrations (only works with AWS SQS for now):

CREATE NOTIFICATION INTEGRATION <integration_name>
  ENABLED = true
  TYPE = QUEUE
  NOTIFICATION_PROVIDER = AWS_SQS
  DIRECTION = OUTBOUND
  AWS_SQS_ARN = '<queue_arn>'
  AWS_SQS_ROLE_ARN = '<iam_role_arn>'

And the new ERROR_INTEGRATION parameter for pipes:

CREATE PIPE <name>
  [ AUTO_INGEST = TRUE | FALSE  ]
  [ AWS_SNS_TOPIC = <string> ]
  ERROR_INTEGRATION = <integration_name>
  AS <copy_statement>

ALTER PIPE <name> SET ERROR_INTEGRATION = <integration_name>;

Test Plan

• acceptance tests

References

• https://docs.snowflake.com/en/LIMITEDACCESS/data-load-snowpipe-notifications.html

[alldoami]

Are you going to implement the error_integration feature? Also I think it'd be better if you rename to error_notification.

[gouline]

Sorry, I was still figuring out how everything works here 😅 . I think I covered everything now? Please let me know if I missed anything, I'm new to this codebase.

The name error_integration comes from the actual parameter set in CREATE PIPE and ALTER PIPE, see in documentation.

[alldoami]

Could you also add a new acceptance test with setting the error_integration 😄

[alldoami]

Also run make docs

[gouline]

Could you also add a new acceptance test with setting the error_integration 😄

Are acceptance tests run on an actual Snowflake account? If so, this might be problematic since the error notifications feature is currently in private preview and Snowflake staff need to enable it for that particular account before setting error_integration parameter would work.

Other stuff is all fixed ✅

[gouline]

Something else I missed: I had thought that notification integration needed for this feature was already implemented, but that was only for AZURE_STORAGE_QUEUE and these Snowpipe error notifications only work on AWS for now. So I added an AWS_SQS implementation and corresponding unit tests.

Notification integrations can also be GCP_PUBSUB, but that's outside the scope of this pull request and I will leave it for somebody using GCP to implement 🙂

[gouline]

Exposed AWS_SQS_EXTERNAL_ID for notification integration, I think that's everything now. Let me know if there's anything else I need to do before this can get merged.

[alldoami]

/ok-to-test sha=b7f0b08

[alldoami]

Run make docs

[github-actions]

Integration tests failure for b7f0b08

[alldoami]

I'm looking into turning on the error notifications for our test account, but it looks like there are some other errors...

[gouline]

Docs re-generated and the two failing tests should now be fixed ✅

[alldoami]

/ok-to-test sha=479f1ee

[github-actions]

Integration tests success for 479f1ee