Value Size Limit

[dc0d]

Is there a hard coded value size limit for badger?

Since badger saves keys and pointer to values, it seems it would not have problems with a large value size - say 2 MB.

[manishrjain]

We haven't set any value size limits. 2mb should be alright.

[kotyara85]

@manishrjain looks like I'm hitting this error - panic: Value with size 2346630 exceeded 1048576 limit. Value
Is there a way to increase?
Thanks

[pandeyshubham25]

@kotyara85 , with the default options, limit on value size is set to 1MB (1048576B). To increase the threshold, please set the value of 'ValueThreshold' parameter in Options to a higher number.
For example, if you want to set it to 5MB you can do it as -

opts := badger.DefaultOptions(path)
opts = opts.WithValueThreshold(5242880)
badger.Open(opts)

Let us know if this works/doesn't work for you, or you need any other details around this.

[kotyara85]

@pandeyshubham25 yeah, tried that already

$ go run main.go
panic: Invalid ValueThreshold, must be less or equal to 1048576

Sounds like a bug

[pandeyshubham25]

@kotyara85 . I managed to look into Badger's code, Open() enforces ValueThreshold to be smaller than maxValueThreshold, a value that is maintained internally and cannot be changed by the client at the moment. This means that you cannot have value sizes in excess of 1MB for now in the LSM tree. You can still insert values of larger size but they would inevitably go to ValueLog.
We can go ahead and look at the implications of having such large values to possibly increase the threshold in future.

[caevv]

This should be re-opened if still an issue. Also interested on the outcome.