wrong size of elements?

[nikooo777]

Hello there,

First of all, thank you so much for this nice implementation of LFUDA! I read your post on medium and I really enjoyed it.

I was testing around with it and I noticed that when passing booleans as values of an element, the size of the value turns out to be 4 bytes which is kinda odd given that a boolean is supposed to be just 1 byte.
I did some digging and I found the snippet of code that calculates the size and I wondered if there was any reason I could not see as to why you did it that way.

		// if the value is binary
		if valBytes, ok := value.([]byte); ok {
			numBytes = float64(len(valBytes))
		} else {
			// otherwise use the default format
			numBytes = float64(len([]byte(fmt.Sprintf("%v", value.(interface{})))))
		}

I put together a quick snippet that would compare your implementation with 2 others I pieced together, one seems to be more accurate than the other, however it still fails on at least one example.

Do you have any opinions on what the real solution should be? I'd be very interested in using the library, however I have to get this right.

https://play.golang.org/p/GdkA4azQqZB

Best regards & stay safe!

Niko

[bparli]

Hi @nikooo777, I'm glad you find this package useful! You found the roughest part of the code and I'm still not sure on a good, general solution. Binary.Size() works in some cases but I didn't find unsafe to work in most. Reflect.TypeOf was actually similar to binary IIRC.

I pushed PR #4 to include binary.Size for the more complex types and booleans. I feel there must be a better way to do this but still don't see it. Would appreciate if you can have a look and let me know what you think

The []interface{}{x} case is an interesting one, but wouldn't be covered by these updates still unfortunately

[nikooo777]

hey @bparli

I checked your PR and it looks like a reasonable first step to take. I wasn't able to find anything better out there, but we can't be the only ones having this problem to solve so eventually we'll get it right :)

If you're curious to see this library in use and how it performs I'd gladly loop you in. I implemented it on a project we use to stream content to users and it basically handles evictions from disk based on the observed requests.

I'm experiencing good results with about 850000 (~1.7TB) items tracked through LFUDA (data on nvme) however it gets a bit more interesting when handling 30'000'000 items (almost 60TB) on disk.

My intentions are to investigate the performance issues i'm noticing and possibly improving your library.

[bparli]

Hey @nikooo777, thats really great to see you're pushing its limits. For now I'll merge that PR but would be happy to hear more about your findings and collaborate on improvements.

[nikooo777]

Nice! I'll update it tomorrow and gather some more data. I'll keep you posted! ;)