Performance, decomposing requests and code generation #72
Comments
|
Here are some ideas: First step would be to allow users to query indexes by themselves. The ids, err := db.Index("User", "Group", "staff")
// signature
Index(bucketName string, fieldName string, value interface{}) ([][]byte, error)The good side of this method is that we don't need to tell what kind of index it is (list vs unique), both will return a list of ids. We could also play with the ids, err := db.From("User").Index("Group", "staff")
// signature
Index(fieldName string, value interface{}) ([][]byte, error) |
|
We could then have a method to fetch all the ids at once. users := make([]User, len(ids))
i := 0
err := db.GetAll("User", ids, func(value []byte, codec storm.Codec) error {
err := codec.Decode(value, &users[i])
if err != nil {
return err
}
i++
return nil
})
// signature
GetAll(bucketName string, ids [][]byte, func([]byte, storm.Codec) error) error |
|
Or we could simply ask the user to provide a pointer to the variable where he wants Storm to decode the value: users := make([]User, len(ids))
err := db.GetAll("User", ids, func(i int) interface{} {
return &users[i]
})
// signature
GetAll(bucketName string, ids [][]byte, func(int) interface{}) error |
|
The remaining reflection is in the |
|
After running some benchmarks, it turns out that removing reflection for reads doesn't speed things up at all: What we can see:
~= Approximately the same speed, sometimes slightly slower, sometimes slightly faster. Now, let's change the Codec and use JSON: Still the same speed, but a lot faster for both tests. Conclusion: The bottleneck is the Codec, not the struct parsing that happens before. Providing a Codec that uses generated code like Protobuf or ffjson would speed things up. PS: |
|
Benchmark runs with benchcmp (before, after) would be easier to understand, but I'm not surprised by the conclusion. |
|
Here is the comparison (i didn't know benchcmp, i like it) |
|
That looks MUCH faster to me...? |
|
Yeah but all i did was changing the default codec in storm.go. The |
|
@asdine what two codecs are you using in the most recent benchcmp results? |
|
@nooproblem |
|
@asdine both tests and benchmarks fail for me when changing to JSON as default codec. I think it would add value to somehow run all tests with both (and I suspect JSON is a better default, too). |
|
Yep, done here: 7aa972a on the branch The failed test is because it assumes that the codec is gob, and tries to decode it with the Here are the results on my current holliday laptop (a ultra slow 200$ Asus C200 chromebook): It seems that the JSON codec is much faster with Storm because |
|
As @bep mentionned, i think we should set the default codec to JSON, it will increase dramatically the read speed and the memory footprint. The downside of using JSON is that the records are no longer ordered in a "natural" way:
So a function like Also, i think we can drop (at least for now) |
Sort by default? I don't think so. I don't think any of the big vendors in the SQL world is doing that. In SQL, if you want a guaranteed order, you'll have to add an |
|
Yeah you are right. I think we can close this one after changing the default codec and remove the |
Storm's primary goal is to be simple to use.It has never been about achieving good performance, that's why reflection is heavily used.
I am not a fan of reflection, it just happened naturally when designing the API.
I sacrified raw speed for ease of use and i am glad i did.
But i believe we can do something to avoid reflection when not needed, and without a lot of work.
Basically, getting one of several records is like this:
The reflection boilerplate is essentially done at the first step, it is necessary so we can collect the following informations:
I think that if we can provide a set of methods that allow the users to manually provide these informations, we could achieve excellent performance.
These methods could also be used internally to simplify some parts of the code.
But the most interesting part is that we may be able to transform current struct declarations that use reflection into using the new methods that we talked about above, at compile time, using
go generateor whatever.The text was updated successfully, but these errors were encountered: