track mongodb _id field so we can attempt to reissue queries #213
Conversation
| func rawDataFromString(s string) interface{} { | ||
| if bson.IsObjectIdHex(s) { | ||
| return bson.ObjectIdHex(s) | ||
| } | ||
| if i, err := strconv.Atoi(s); err != nil { | ||
| if i, err := strconv.Atoi(s); err == nil { |
There was a problem hiding this comment.
this was just bad logic
| func rawDataFromString(s string) interface{} { | ||
| if bson.IsObjectIdHex(s) { | ||
| return bson.ObjectIdHex(s) | ||
| } | ||
| if i, err := strconv.Atoi(s); err != nil { | ||
| if i, err := strconv.Atoi(s); err == nil { |
|
|
||
| func sortable(id interface{}) bool { | ||
| switch id.(type) { | ||
| case bson.ObjectId, string, float64, time.Time: |
There was a problem hiding this comment.
Nice, all there any other sortable types _id can be? maybe int family?
There was a problem hiding this comment.
from the testing I did, int's seem to always come back as float64 but I'll throw int64 in there just to be safe
| @@ -359,35 +359,36 @@ func (m *MongoDB) catData() error { | |||
| } else if match := m.collectionMatch.MatchString(collection); !match { | |||
| continue | |||
| } | |||
| query := bson.M{} | |||
| // lastID := "" | |||
| canReissueQuery := m.requeryable(collection) | |||
There was a problem hiding this comment.
Hmm, this should be okay to cache. Hopefully people aren't messing with their indices while transporter is running :scary:
There was a problem hiding this comment.
yea, I wouldn't expect people to be messing with the primary _id index much to begin with much less while trying to perform a data transfer .
|
@trinchan good point on incremental addition of tests for the helper function. I plan on getting another PR in soon to lay a foundation for good e2e tests for adaptors. |
jipperinbham
force-pushed
the
track-id
branch
from
f03d3ac
to
03ab954
Compare
This change brings back the ability to re-issue a query after an err has been encountered while iterating a collection. For now, we will only attempt to perform the retry if the
_idfield is properly indexed is one ofbson.ObjectId, string, float64, time.Time.