Add redis.Scan() to scan results from redis maps into structs. #1631

knadh · 2021-01-28T05:40:36Z

The package uses reflection to decode default types (int, string etc.) from Redis map results (key-value pair sequences) into
struct fields where the fields are matched to Redis keys by tags.

Similar to how encoding/json allows custom decoders usingUnmarshalJSON(), the package supports decoding of arbitrary
types into struct fields by defining a Decode(string) errorfunction on types.

The field/type spec of every struct that's passed to Scan() is cached in the package so that subsequent scans avoid iteration
and reflection of the struct's fields.

Issue: #1603

decoder.go

vmihailenco · 2021-01-28T09:13:18Z

👍

I've left some comments - hopefully it is not too much :) Also please set "Allow edits from maintainers" on the PR so I can add commits...

Let's move the code to internal/hscan package for now so all types are private. And the public API could look like err := rdb.HGetAll(ctx, "key").Scan(&strct).

The package uses reflection to decode default types (int, string etc.) from Redis map results (key-value pair sequences) into struct fields where the fields are matched to Redis keys by tags. Similar to how `encoding/json` allows custom decoders using `UnmarshalJSON()`, the package supports decoding of arbitrary types into struct fields by defining a `Decode(string) error` function on types. The field/type spec of every struct that's passed to Scan() is cached in the package so that subsequent scans avoid iteration and reflection of the struct's fields.

knadh · 2021-01-28T12:37:12Z

Incorporated all the changes you suggested and moved the package to internal/hscan.

PS: Did a force push because it was mostly hygiene fixes.

internal/hscan/structmap.go

vmihailenco · 2021-01-29T10:08:49Z

@knadh I've pushed few small changes, but this looks good already.

Could you add SliceCmd.Scan and a test (or two :)?

internal/hscan/structmap.go

knadh · 2021-01-29T10:12:00Z

Could you add SliceCmd.Scan and a test (or two :)?

Yep, will push this.

knadh · 2021-02-02T07:36:20Z

Could you add SliceCmd.Scan and a test (or two :)?

Added here: a4144ea

vmihailenco · 2021-02-02T07:53:20Z

internal/hscan/structmap.go

+		tag := f.Tag.Get(fieldTag)
+		if tag == "" || tag == "-" {
+			continue
+		}


It looks we are ignoring fields that don't have redis:"something" tag. Is that intentional?

Usually we are supposed to generate default name from Go field name and only ignore the field when redis:"-". What do you think?

https://pkg.go.dev/github.com/codemodus/kace could be used to generate default name.

this parser is too simple....can only do the most basic work, it needs to continue to improve its functions,
i can continue to improve it this week,,,if needed

It looks we are ignoring fields that don't have redis:"something" tag. Is that intentional?

Yeah. Explicitly defining what's to be scanned is better (fewer gotchas) in my opinion. Opting in rather than opting out with "-".

this parser is too simple....can only do the most basic work, it needs to continue to improve its functions,

It only handles primitive types. The right way to extend it with the Decode() interface which was in the original PR. I hope @vmihailenco has plans to introduce that functionality soon!

vmihailenco · 2021-02-02T08:04:19Z

Could you add one end-to-end test or maybe even an example? That will help people to get started

rdb.HSet(..)
rdb.HSet(...)

rdb.HGetAll(ctx, "key").Scan(&dst)

knadh · 2021-02-02T09:27:03Z

I just realised, StringStringMapCmd will also need to support Scan() for HGetAll() to work. Okay to add?

command.go

vmihailenco · 2021-02-03T10:50:29Z

@knadh I've pushed a change with StructValue so StringStringMapCmd.Scan can be slightly more efficient. I think this PR is ready to be merged if you are happy with that change. It will be part of v8 but marked unstable so we can change it based on the feedback.

vmihailenco · 2021-02-03T10:51:38Z

internal/hscan/hscan_test.go

+			Float  float32 `redis:"float"`
+			Bool   bool    `redis:"bool"`
+		}
+		var d2 data2


Just FYI this can be written as

var d2 struct { String string `redis:"string"` Bytes []byte `redis:"byte"` Int int `redis:"int"` Uint uint `redis:"uint"` Float float32 `redis:"float"` Bool bool `redis:"bool"` }

monkey92t · 2021-02-03T11:04:18Z

missing test for mget command, we should complete it.
and...if it is 386 platform, there may be ERROR: "strconv.ParseInt: parsing "1234567890123456789": value out of range".
reflect.Int64、reflect.Uint64、reflect.Float64,
but I’m not sure if we still need to be compatible with the 386 platform

redis 127.0.0.1:6379>> set key 1234567890123456789

type Temp struct {
	Key int64 `redis:"key"`
}

GOARCH=386 go test it should run normally.

knadh · 2021-02-03T11:44:00Z

missing test for mget command, we should complete it.

Added test https://github.com/knadh/redis/commit/f8a546b4820de9e52fe0d244e24dca155688abef. @monkey92t I guess you could add platform specific parsing.

@knadh I've pushed a change with StructValue so StringStringMapCmd.Scan can be slightly more efficient. I think this PR is ready to be merged if you are happy with that change. It will be part of v8 but marked unstable so we can change it based on the feedback.

@vmihailenco looks fine. Please go ahead.

vmihailenco · 2021-02-04T07:48:57Z

🥳

mdelclaro · 2021-02-25T15:24:22Z

Hello @vmihailenco @knadh

What if in this case I have a slice like []interface{} instead of an interface only? I'm getting this redis.Scan(non-struct *[]models.Price)

I see that in the code you check for if v.Kind() != reflect.Struct

I need it to be a slice since the result I'm getting from MGet is a slice.

Any ideias on what I'm doing wrong?

monkey92t · 2021-02-25T15:47:35Z

Hello,

What if in this case I have a slice like []interface{} instead of an interface only? I'm getting this redis.Scan(non-struct *[]models.Price)

I see that in the code you check for if v.Kind() != reflect.Struct

I need it to be a slice since the result I'm getting from MGet is a slice.

Any ideias on what I'm doing wrong?

under normal circumstances, the data format of redis is as follows:

127.0.0.1:6379>>set id 100
127.0.0.1:6379>>set name monkey
127.0.0.1:6379>>set age 20

127.0.0.1:6379>>mget id name age
//{"id": 100, "name": "monkey", "age": 20}

//golang
type Per struct {
	ID   int    `redis:"id"`
	Name string `redis:"name"`
	Age  int    `redis:"age"`
}
p := &Per{}
err := redis.Mget(ctx, "id", "name", "age").Scan(p)

// p.ID = 100
// p.Name = monkey
// p.Age = 20

i don’t know what you mean by []interface, if i don’t understand correctly, you can add an example to describe your problem in detail.

mdelclaro · 2021-02-25T18:11:07Z

Hello,
What if in this case I have a slice like []interface{} instead of an interface only? I'm getting this redis.Scan(non-struct *[]models.Price)
I see that in the code you check for if v.Kind() != reflect.Struct
I need it to be a slice since the result I'm getting from MGet is a slice.
Any ideias on what I'm doing wrong?

under normal circumstances, the data format of redis is as follows:
127.0.0.1:6379>>set id 100
127.0.0.1:6379>>set name monkey
127.0.0.1:6379>>set age 20

127.0.0.1:6379>>mget id name age
//{"id": 100, "name": "monkey", "age": 20}

//golang
type Per struct {
	ID   int    `redis:"id"`
	Name string `redis:"name"`
	Age  int    `redis:"age"`
}
p := &Per{}
err := redis.Mget(ctx, "id", "name", "age").Scan(p)

// p.ID = 100
// p.Name = monkey
// p.Age = 20
i don’t know what you mean by []interface, if i don’t understand correctly, you can add an example to describe your problem in detail.

I have hundreds of keys like price/tokenX-tokenY, so for what I see I should create a struct with all the keys, which would not be possible. And each value is a JSON structure.

The response from MGet is something like this:

[ {"id":1,"base":"X","quote":"Y", "price": 1} ]

I'm a little lost here on what I have to do :')

mdelclaro · 2021-02-25T19:36:14Z

Hello,
What if in this case I have a slice like []interface{} instead of an interface only? I'm getting this redis.Scan(non-struct *[]models.Price)
I see that in the code you check for if v.Kind() != reflect.Struct
I need it to be a slice since the result I'm getting from MGet is a slice.
Any ideias on what I'm doing wrong?

under normal circumstances, the data format of redis is as follows:
127.0.0.1:6379>>set id 100
127.0.0.1:6379>>set name monkey
127.0.0.1:6379>>set age 20

127.0.0.1:6379>>mget id name age
//{"id": 100, "name": "monkey", "age": 20}

//golang
type Per struct {
	ID   int    `redis:"id"`
	Name string `redis:"name"`
	Age  int    `redis:"age"`
}
p := &Per{}
err := redis.Mget(ctx, "id", "name", "age").Scan(p)

// p.ID = 100
// p.Name = monkey
// p.Age = 20
i don’t know what you mean by []interface, if i don’t understand correctly, you can add an example to describe your problem in detail.
I have hundreds of keys like price/tokenX-tokenY, so for what I see I should create a struct with all the keys, which would not be possible. And each value is a JSON structure.

The response from MGet is something like this:

[ {"id":1,"base":"X","quote":"Y", "price": 1} ]

I'm a little lost here on what I have to do :')

I resolved it by doing the following:

res := Cli.MGet(ctx, keys...)

var prices []models.Price

str := fmt.Sprintf("%+v", res.Val())
rawIn := json.RawMessage(str)

bytes, err := rawIn.MarshalJSON()

json.Unmarshal(bytes, &prices)

Thanks!

knadh · 2021-02-26T03:41:28Z

@mdelclaro The Scan() function is specifically for scanning Redis values by their keys into structs. This is useful for MGET, HGETALL etc. In your example, you are pulling JSON strings as values from Redis and scanning keys of JSON strings, not keys from Redis. So, you indeed have to use json.Unmarshal.

csdenboer · 2021-07-01T06:17:11Z

Scan doesn’t seem to work on nested structures: #1806. Based on the comments in this PR, it seems that “the package supports decoding of arbitrary
types into struct fields by defining a Decode(string) error function on types” was not actually implemented. Is this correct? If yes, will it be implemented anytime soon? Is there any other way to make this work?

ysweid · 2021-09-11T06:51:17Z

Decoding time.Time could be a big win.

haijianyang · 2021-12-28T03:52:51Z

cannot scan redis.result 2021-12-28T11:31:45.611441+08:00 into struct field UpdatedAt of type time.Time, error-redis.Scan(unsupported time.Time)

Hi guys, when scan will support time.Time?

knadh · 2021-12-28T04:07:21Z

The issue with time.Time is the format to parse. Since 2021-12-28T11:31:45.611441+08:00 is the default format that's stringified, maybe we can hardcode scanning for just this one format and document that. @vmihailenco what do you think?

errpunk · 2022-12-12T09:56:05Z

ping,
really need this

monkey92t · 2022-12-24T14:30:07Z

Add Scanner interface, See #2317

knadh mentioned this pull request Jan 28, 2021

PR propsal: Scanning results and maps to struct #1603

Closed

vmihailenco reviewed Jan 28, 2021

View reviewed changes