New sink: Oracle Cloud Infrastructure 'oci_object_storage'

[CarlDownsOracle]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Background

• We have a customer that has standardized on Vector for their observability data collection needs.
• Customer is moving AWS Kubernetes and Compute based workloads to Oracle Cloud Infrastructure (OCI).
• Customer use case calls for archival of logs and metrics to OCI object storage.
• Vector has several sink implementations, including aws_s3 sink.
• https://vector.dev/docs/reference/configuration/sinks/aws_s3/

Attempted Solutions

Success with Vector

• The aws_s3 sink works with OCI’s S3 compatible API using OCI auth credentials.

Customer Ask

• Customer is asking if we can support an analog to the auth.assume_role feature in OCI because they are reticent to include any credentials in configuration files.
• https://vector.dev/docs/reference/configuration/sinks/aws_s3/#auth.assume_role
• OCI’s analog is something called an instance principal where Compute-based OCI API calls are authorized without passing in credentials.
• https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm

Proposal

We are looking to see if there is an implementation path for an OCI-specific sink that can take advantage of the instance principal mechanism.

References

No response

Version

No response

[CarlDownsOracle]

Hi @jszwedko, just checking in to see what your thoughts are on this request.

[jszwedko]

Hi @CarlDownsOracle !

The request makes sense. I saw another recent issue, #16665 , for making it easier to use the aws_s3 sink with OCI object storage since it has an AWS S3 compatible endpoint, but I see your note here about wanting to support OCI's authentication mechanisms, for which a specialized sink makes sense.

We recently added a sink that leverages OpenDAL -- an abstraction over cloud storage engines. Adding OCI object storage support there could make it easier for Vector to support in the future (apache/opendal#377 seems to be tracking that over there).

Thanks for filing!

[CarlDownsOracle]

Thanks Jesse ! I’ll look at the OpenDAL project … thanks for passing the ticket over as well! Regards, Carl [signature_1585353333] Carl Downs Principal Solution Architect Program Management Office From: Jesse Szwedko ***@***.***> Date: Thursday, March 16, 2023 at 11:45 AM To: vectordotdev/vector ***@***.***> Cc: Carl Downs ***@***.***>, Mention ***@***.***> Subject: [External] : Re: [vectordotdev/vector] New sink: Oracle Cloud Infrastructure 'oci_object_storage' (Issue #16766) Hi @CarlDownsOracle<https://urldefense.com/v3/__https:/github.com/CarlDownsOracle__;!!ACWV5N9M2RV99hQ!MW0-l0FH98jEDnnTAzNLT_B1R2DNX7QCGPGgPx8bbrmNaKsDgXOrMD8JOULXhBW3LtyIvufHBjk0z_t1G_69wZwr2Q$> ! The request makes sense. I saw another recent issue, #16665<https://urldefense.com/v3/__https:/github.com/vectordotdev/vector/issues/16665__;!!ACWV5N9M2RV99hQ!MW0-l0FH98jEDnnTAzNLT_B1R2DNX7QCGPGgPx8bbrmNaKsDgXOrMD8JOULXhBW3LtyIvufHBjk0z_t1G_68FM0lCw$> , for making it easier to use the aws_s3 sink with OCI object storage since it has an AWS S3 compatible endpoint, but I see your note here about wanting to support OCI's authentication mechanisms, for which a specialized sink makes sense. We recently added a sink that leverages OpenDAL<https://urldefense.com/v3/__https:/github.com/apache/incubator-opendal__;!!ACWV5N9M2RV99hQ!MW0-l0FH98jEDnnTAzNLT_B1R2DNX7QCGPGgPx8bbrmNaKsDgXOrMD8JOULXhBW3LtyIvufHBjk0z_t1G_4rARa7dA$> -- an abstraction over cloud storage engines. Adding OCI object storage support there could make it easier for Vector to support in the future (apache/opendal#377<https://urldefense.com/v3/__https:/github.com/apache/incubator-opendal/issues/377__;!!ACWV5N9M2RV99hQ!MW0-l0FH98jEDnnTAzNLT_B1R2DNX7QCGPGgPx8bbrmNaKsDgXOrMD8JOULXhBW3LtyIvufHBjk0z_t1G_6la4M6Kw$> seems to be tracking that over there). Thanks for filing! — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/vectordotdev/vector/issues/16766*issuecomment-1472332866__;Iw!!ACWV5N9M2RV99hQ!MW0-l0FH98jEDnnTAzNLT_B1R2DNX7QCGPGgPx8bbrmNaKsDgXOrMD8JOULXhBW3LtyIvufHBjk0z_t1G_7SZ7jNmg$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AOTLLTSONFAXRAXM257KWDLW4M7T3ANCNFSM6AAAAAAVW5P6NY__;!!ACWV5N9M2RV99hQ!MW0-l0FH98jEDnnTAzNLT_B1R2DNX7QCGPGgPx8bbrmNaKsDgXOrMD8JOULXhBW3LtyIvufHBjk0z_t1G_4h8g1MVA$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Xuanwo]

Challenge accepted! We (at OpenDAL) will soon implement support for OCI.

[CarlDownsOracle]

Thanks @Xuanwo ! Please reach out if you have any questions ... or we can assist in some way.

[Xuanwo]

Thanks @Xuanwo ! Please reach out if you have any questions ... or we can assist in some way.

I believe that the Always Free Tier will suffice for our development and integration testing needs. I'll let you know if we encounter any further issues. Thank you in advance.

[CarlDownsOracle]

Yes the Always Free Tier should be sufficient. You probably already have this:

https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm

Thank you!

[CarlDownsOracle]

Hello @Xuanwo happy Friday. Just checking in to see how things are going.

[Xuanwo]

Hello @Xuanwo happy Friday. Just checking in to see how things are going.

Happy Friday to you too! This week, I am working on the implementation. However, there is a potential block as I am unable to register an OCI account which is necessary for integrating with real services. Despite submitting a ticket 10 days ago (ticket number 230321-001192, account oracle@xuanwo.io), I have not received any response yet. Would it be possible for you to assist me in resolving this issue?

[CarlDownsOracle]

Let me see if I can help. That ticket number ... may I ask in which system? Can't find it in My Oracle Support.

[Xuanwo]

Let me see if I can help. That ticket number ... may I ask in which system? Can't find it in My Oracle Support.

I think it called Oracle Customer Service, sent by oraclehelp_ww@oracle.com

[CarlDownsOracle]

Hi Xuanwo – I’m unable to locate your request. If you want to DM or email me directly, I can help you. Regards, Carl [signature_117748667] Carl Downs Principal Solution Architect Program Management Office From: Xuanwo ***@***.***> Date: Friday, March 31, 2023 at 12:47 PM To: vectordotdev/vector ***@***.***> Cc: Carl Downs ***@***.***>, Mention ***@***.***> Subject: [External] : Re: [vectordotdev/vector] New sink: Oracle Cloud Infrastructure 'oci_object_storage' (Issue #16766) Let me see if I can help. That ticket number ... may I ask in which system? Can't find it in My Oracle Support. I think it called Oracle Customer Service, sent by ***@***.*** — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/vectordotdev/vector/issues/16766*issuecomment-1492364789__;Iw!!ACWV5N9M2RV99hQ!JlCeGCOTja4dQEa1UQVt9XhC4YWGoxT-Tt43MuNJUYZ027N4Tqt6MR2kFwuTw-djHOYYAiDkFiup8TvKUcGeCzjCzg$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AOTLLTVIZQEJG4U67ZQEXMDW64KARANCNFSM6AAAAAAVW5P6NY__;!!ACWV5N9M2RV99hQ!JlCeGCOTja4dQEa1UQVt9XhC4YWGoxT-Tt43MuNJUYZ027N4Tqt6MR2kFwuTw-djHOYYAiDkFiup8TvKUcH5OdTQrA$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Xuanwo]

Hi, this issue is blocked by apache/opendal#2563

All of our team members were unable to obtain an Oracle Cloud account, even after contacting the seller through formal channels.