Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Object Storage #38

Open
perriea opened this issue May 28, 2019 · 14 comments

Comments

@perriea
Copy link

commented May 28, 2019

Is it expected that the project will support object storage natively like Thanos or Cortex (in futur) ?
Object storage like S3 or GCS.

@valyala valyala added the enhancement label May 28, 2019

@valyala

This comment has been minimized.

Copy link
Contributor

commented May 28, 2019

Thanks for the idea, @perriea ! Could you provide more context where this may be necessary?

Probably it is possible to point VictoriaMetrics (via -storageDataPath) to a directory mounted from S3 or GCS via FUSE wrappers such as gcsfuse or s3fs-fuse. Could you try such setup and report results here?

@MaxDiOrio

This comment has been minimized.

Copy link

commented May 31, 2019

Thanos allows storing of metric data directly in S3 or GCS using Minio.

I'd love to see snapshots be able to be written directly to S3 or GCS. One step backup and truncation of data.

@perriea

This comment has been minimized.

Copy link
Author

commented May 31, 2019

Yes, exactly @MaxDiOrio that's what I'm looking for because I'd like to test this project in a Kubernetes or with (high) scalability context without having to do stateful.
However I will test what I recommended @valyala although I know very well that the system fuse is not crazy on the performances.

@valyala

This comment has been minimized.

Copy link
Contributor

commented May 31, 2019

I'd love to see snapshots be able to be written directly to S3 or GCS. One step backup and truncation of data.

Unfortunately this isn't easy as it sounds. Automatic uploading of snapshots to GCS and S3 may be interrupted due to various failures:

  • network connectivity issues
  • intermittent object store maintenance / outage

The probability of errors increases with the size of snapshot files. VictoriaMetrics may produce multi-TB files in snapshots, which result in very high probability of errors during automatic snapshot uploading. How to handle such errors? There are various approaches for errors handling and uploading strategies with different pros & cons:

  • Just retry file upload from the beginning. This may be huge waste of time for multi-TB files.
  • Resume the upload from the failed place if this is supported by the object store.
  • Split big files in smaller chunks and upload them separately with per-chunk retries on errors. During the recovery merge these chunks into the original files.
  • Archive the snapshot into a single file, then split it into moderately sized chunks and upload them in parallel to object store in order to improve upload performance.
  • Perform incremental backup, i.e. upload only new files and delete missing files from the object store. This is possible because of LSM tree used by VictoriaMetrics.
  • Your approach.

There are other approaches exist. Each approach has certain users, but there is no "silver bullet" approach for snapshot uploading, which would suit all the users. So it is highly unlikely VictoriaMetrics will automatically upload snapshots to object store in the future.

@valyala

This comment has been minimized.

Copy link
Contributor

commented May 31, 2019

Yes, exactly @MaxDiOrio that's what I'm looking for because I'd like to test this project in a Kubernetes or with (high) scalability context without having to do stateful.

It is better to use persistent volumes backed by replicated durable disks for Kubernetes such as Google Compute Engine persistent disks - see these docs for details. VictoriaMetrics is optimized for HDD-backed persistent disks, so there is no much sense in using more expensive SSD-backed persistent disks.

@antoniobeyah

This comment has been minimized.

Copy link

commented May 31, 2019

Agreed with this request. Using persistent volumes within Kubernetes in AWS typically defaults to using EBS which is tied to an availability zone which is a challenge when you have a k8s cluster that spans multiple availability zones. You cannot use the same EBS volume across availability zones so it creates issues when your cluster is in an autoscaling group.

@MaxDiOrio

This comment has been minimized.

Copy link

commented May 31, 2019

This is what multipart S3 uploads help solve. I do understand the challenge though. I'd picture doing an automated monthly backup, which should keep the size more manageable.

@MaxDiOrio

This comment has been minimized.

Copy link

commented May 31, 2019

Maybe do a daily snapshot/archive to cloud storage for small file size with a monthly purge? Definitely check how Thanos is doing this. They are storing and querying directly to cloud storage. Clearly it's not going to be as performant as local storage but it would be a good option.

It could also be a good option for those of us with on-prem slow storage that we can use with minio as object storage. Still local and fast but not on our primary, expensive ssd storage.

It would be great to archive off but still be able to query the archived data like Thanos.

If this can be done without the complexity of Thanos that would be amazing. I spent a full week trying to get the Thanos and sidecar pieces set up but had too many issues. That drove me to Victoria Metrics as a probable solution.

@valyala

This comment has been minimized.

Copy link
Contributor

commented May 31, 2019

It shouldn't be too hard to store VictoriaMetrics' data in object storage if the storage supports the following operations:

  1. Streaming writing to a file without limits on file size. Resume streaming writing after network error must be supported.
  2. Fast concurrent reading of arbitrary parts of the file.

The main issue with object storage is network errors. They reduce reliability and availability comparing to much more reliable block storage such as GCE disks, EBS or local disks. Currently VictoriaMetrics assumes that it works with reliable storage (aka filesystem), so it reports the first occurred error related to storage and exits instead of trying to recover from such errors (it is unclear how to recover from these errors). When working with object storage, such errors must be handled gracefully, since they are usually expected in object storage world. This is one of the reasons why Thanos has not so good reliability comparing to VictoriaMetrics.

@valyala

This comment has been minimized.

Copy link
Contributor

commented Jun 6, 2019

There is yet another approach for adding ability to use Object Storage as a store for VictoriaMetrics data: to write a thin layer between file API and Object Storage API, which will split big files into smaller chunks and upload them to Object Storage as separate files. This should alleviate network connectivity issues outlined above and will improve upload retrying speed.

@perriea

This comment has been minimized.

Copy link
Author

commented Jun 7, 2019

It seems like a good idea. How do you see this feature in VictoriaMetrics ? Native functionality ? Plugin ?
I ask because there are different protocols (S3, GCS, Azure Blob storage for the main ones).

@valyala

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2019

There should be native functionality implemented for each supported protocol. Probably https://github.com/minio/minio-go could simplify the integration.

@MaxDiOrio

This comment has been minimized.

Copy link

commented Jun 7, 2019

Yes...native is probably the way to go.

@Serrvosky

This comment has been minimized.

Copy link

commented Aug 1, 2019

That solution could be awesome and a massive improve to Victoria Metrics. Can bring a must have feature with the simple approach like Victoria Metrics got us used.

It would be nice to have a flag like timeToSnapshot, where Vitoria Metric's storage nodes can read an interval of time to make a snapshots and send it to the cloud, leaving nothing more that a proper logs .
If something happen there are a lot of log's aggregation and processing tools (like datadog, elastiksearch, loki+grafana) where users can check error logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.