Some of my posts don't decode

[bradfitz]

gosky list --all works on most of my repo, but some values don't parse:

% gosky list --all --values $(gosky handle resolve bradfitz.com) | grep -A 1 panic
panic:  unreachable
bad blob: a5652474797065766170702e62736b792e6163746f722e70726f66696c6566617661746172a463726566d82a58250001551220cb04502240be648abdc77c7e9a5ef1750a2b0943ab6bb5f6b4fc7a16dc59092a6473697a651a0001afd965247479706564626c6f62686d696d65547970656a696d6167652f6a7065676662616e6e6572a463726566d82a5825000155122050509cd8bd1f3e4834eb5d369c712d6bd84475a795d5f309fbd26dabc07476386473697a651a0002b1fc65247479706564626c6f62686d696d65547970656a696d6167652f6a7065676b6465736372697074696f6e78b54920646f20636f6d7075746572732e204d61727269656420746f20407261646b61742e6669747a7061742e636f6d2e205468726565206b6964732e20586f6f676c65722e200a476f202823676f6c616e6729207465616d20323031307e323032302e204d616465204c6976654a6f75726e616c2c204f70656e49442c206d656d6361636865642e2043757272656e746c79206174205461696c7363616c65206d616b696e672057697265477561726420656173792e6b646973706c61794e616d657042726164204669747a7061747269636b
--
panic:  unreachable
bad blob: a5647465787478254c6f6f6b73206c696b6520746869732077617320796f757220666c696768743f2046756e21652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a58250001551220773ea1e3c22c394429a71709b216ded26d0c939890351ddd37bbdb45dee5ad306473697a651a000f140065247479706564626c6f62686d696d65547970656a696d6167652f6a706567657265706c79a264726f6f74a263636964783b62616679726569673277653373676d3663653278356e666a75717a6171327169653571666c7963366b7068777a376368377535736632756366666163757269784661743a2f2f6469643a706c633a6966327475673562756335613363727a32643269323469332f6170702e62736b792e666565642e706f73742f336a74713371376235657a326d66706172656e74a263636964783b62616679726569673277653373676d3663653278356e666a75717a6171327169653571666c7963366b7068777a376368377535736632756366666163757269784661743a2f2f6469643a706c633a6966327475673562756335613363727a32643269323469332f6170702e62736b792e666565642e706f73742f336a74713371376235657a326d696372656174656441747818323032332d30342d32345431373a31383a32332e3739385a
--
panic:  unreachable
bad blob: a4647465787464f09f9299652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a5825000155122055fcd00bd248cdc46dfed852b93aab637706e8daf716c43504da1dc02ad71ed96473697a651a000d4fa565247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32345432323a30363a30312e3738355a
--
panic:  unreachable
bad blob: a56474657874787e4f682c206e6963652c206d7920667269656e642040627261646669747a2e69732e74657374696e672e686f772e62656175746966756c2e6f662e612e757365726e616d652e796f752e63616e2e686176652e302e302e662e652e622e662e322e302e362e322e6970362e61727061206a6f696e656420426c7565736b7921652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a58250001551220526572a012f96be8743f6388062b9127a0fc332b8f461ce29c85cd42a7e32d636473697a651a0004e8e165247479706564626c6f62686d696d65547970656a696d6167652f6a7065676666616365747381a3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e64186e696279746553746172741468666561747572657381a26364696478206469643a706c633a736637786a68736a3461666368326769366e743633636c65652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6e696372656174656441747818323032332d30342d32355430333a35303a31312e3731315a
--
panic:  unreachable
bad blob: a5647465787478b45475726e73206f7574206d792077696665206f66206e6561726c7920736576656e2079656172732c20407261646b61742e6669747a7061742e636f6d2c206469646e2774206b6e6f772074686174206d7920706572736f6e616c207765627369746520697320627261646669747a2e636f6d2e0a0a5468616e6b7320426c7565736b7920666f72206c657474696e67207573206c6561726e206d6f72652061626f75742065616368206f746865722120f09f9882652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a58250001551220e71409e2ab359e01241ccc7705b84ecac6ff638e11e071ed8cbd7211e61cf6516473697a651a0001362f65247479706564626c6f62686d696d65547970656a696d6167652f6a7065676666616365747382a3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e64183c69627974655374617274182968666561747572657381a26364696478206469643a706c633a6f6b6b33656b7575736b7062736476777972616e72726c71652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6ea265696e646578a26762797465456e64187269627974655374617274186668666561747572657381a2637572697468747470733a2f2f627261646669747a2e636f6d652474797065781c6170702e62736b792e72696368746578742e6661636574236c696e6b696372656174656441747818323032332d30342d32355431373a35393a34372e3739335a
--
panic:  unreachable
bad blob: a4647465787468f09f9299f09f8cb8652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a58250001551220ee70559a326bc006796f3371d9dece0f31c318bf45569ad87d1445cbb9b79d486473697a651a000ee56a65247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32355432303a35353a30382e3538395a
--
panic:  unreachable
bad blob: a564746578747873546f64617920406361747a6b6f726e2e62736b792e736f6369616c20706f73746564207468697320696e206f6e65206f66206f757220766172696f757320536c61636b2073686974706f7374696e67206368616e6e656c7320617420407461696c7363616c652e636f6d202e2e2e20f09f9882652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512205b3dbac27e31007345f2b54ca68dc5b6430a245a5a814c60494c59992318cf296473697a651a0006220365247479706564626c6f62686d696d65547970656a696d6167652f6a7065676666616365747382a3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e64181b696279746553746172740668666561747572657381a26364696478206469643a706c633a7a6e6f71776568777369616563656f757534747471366277652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6ea3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e64186a69627974655374617274185c68666561747572657381a26364696478206469643a706c633a746b633378697961346b37757978336a7365693362703635652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6e696372656174656441747818323032332d30342d32365431383a34393a30392e3833375a
--
panic:  unreachable
bad blob: a46474657874781a4f682c20746865206a6f7973206f6620696e66616e74732e2e2e652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512202bf480f10f5fd233413a06177473f6c4d2d6ea40a8bfbf9122b675067a24945d6473697a651a000d21f565247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32375431373a30393a31322e3038345a
--
panic:  unreachable
bad blob: a4647465787479012e4920686164206120636f6e74726163746f72206f757420646f696e672061206d61696e74656e616e6365207669736974206f6e206f757220686561742070756d70732e2048652068616420746f20676f207468726f756768206d79206f666669636520746f2067657420746f2074686520726f6f6620756e697420616e6420667265616b6564206f7574207768656e20686520736177206d79206465736b2c20667265657a696e6720616e642073746172696e67206174206d652c2061736b696e673a0a0a2257687920646f20796f7520686176652061206b6e696665206f6e20796f7572206465736b3f213f220a0a492068616420746f206578706c61696e20776861742061206c6574746572206f70656e6572206973202620686f7720697420776f726b65642e20f09f9882652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512201d115053c4bf8de9fc6bb243a5b23e82ac20d4f01dfce4ae31a71a0570752b3b6473697a651a000e383a65247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32375431383a33373a34332e3735315a
--
panic:  unreachable
bad blob: a4647465787478d1496620796f75206e65656420746f207761726e206d6520696e20616476616e6365207768656e20796f752068616e64206d652074686520515220636f6465206d656e7520746861742069742070726f6261626c7920776f6e2774207363616e2c206d6179626520697427732074696d6520666f72206e657720515220636f646573207468617420617265206d6f726520626f72696e673f0a0a427574206f6e207468652062726967687420736964652c207468697320776561746865722120416e64206265657221204f7574736964652e652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512202ba9760cd191f411c3a8f31700c4ced2cd7a0f61a4e3e0d0c74488b05b3732aa6473697a651a000eecbb65247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32375432323a32313a32372e3036335a
--
panic:  unreachable
bad blob: a5647465787479012c49207361772070656f706c6520686572652064697363757373696e6720656e6a6f79696e67206c696b696e6720227468697273742220706f737473206f6e207768696c65206c696b65732061726520707269766174652e0a0a4265206361726566756c2e2054686579277265206e6f742e20546865207765622f61707073206a75737420646f6e27742073686f7720796f752c206275742069742773207075626c69632e0a0a652e672e2048657265206172652040616e696c646173682e636f6d2773206c696b65732e204c6173742055524c2069730a0a68747470733a2f2f73746167696e672e62736b792e6170702f70726f66696c652f6469643a706c633a6b646a6d336d7776627662766171746236723369743571712f706f73742f336a75666d7276677069753270652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512200a15cfd7eb305eef376e743752c05754c6bbcabf28c801aac0d347dc49cd067e6473697a651a000ba21f65247479706564626c6f62686d696d65547970656a696d6167652f6a7065676666616365747382a3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e6418c16962797465537461727418b468666561747572657381a26364696478206469643a706c633a73673265326b71647364703271327a6c3434747861646270652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6ea265696e646578a26762797465456e6419012c6962797465537461727418d868666561747572657381a263757269785468747470733a2f2f73746167696e672e62736b792e6170702f70726f66696c652f6469643a706c633a6b646a6d336d7776627662766171746236723369743571712f706f73742f336a75666d7276677069753270652474797065781c6170702e62736b792e72696368746578742e6661636574236c696e6b696372656174656441747818323032332d30342d32385430333a32383a32362e3832355a
--
panic:  unreachable
bad blob: a5647465787460652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512202751a5b99841ef99b3b195e750cb2f6e8b8f47b12f5b99781a09326991f30b076473697a651a000142b465247479706564626c6f62686d696d65547970656a696d6167652f6a706567657265706c79a264726f6f74a263636964783b6261667972656968716b62707236796171376a6c617a666637636763336863326468706a6b616b79677765723362657a676f7869786d763567356163757269784661743a2f2f6469643a706c633a377232667933623475376d6d6e68676264786e666c6f76762f6170702e62736b792e666565642e706f73742f336a75666f636a6763716e327066706172656e74a263636964783b626166797265696470656a3261736b737566716f79647668787736796a66376d33756c71616336787a6174666177666a696e377970356c6977336163757269784661743a2f2f6469643a706c633a3634727976757271777a72366c6a6e3576376c776e696e682f6170702e62736b792e666565642e706f73742f336a75666f6b32656468353270696372656174656441747818323032332d30342d32385430333a33353a32332e3136385a

/cc @whyrusleeping @bnewbold

[bradfitz]

(I haven't looked into what the panic is; just filing before I forget.)

[bnewbold]

I will need to look, but off the top of my head:

• this known issue with some posts not being output correctly (probably not a parse issue though)
• a month ago, the repo MST tree would be "complete" based on spidering CID links out of DAG-CBOR records. aka, every target of a CID would be in the repo. during a lexicon refactor, we now make "blobs" (eg, images) contain CID links. the blobs are not "in" the repo, they are stored separately. there is no current way to get a CAR file with repo and all the blobs as blocks. so if this code is still spidering it will fail. the ipfs CLI tool chokes in this particular way on newer records

[bradfitz]

Removing the recover, here's one of the panic stacks:

app.bsky.actor.profile/self
panic: unreachable

goroutine 1 [running]:
github.com/polydawn/refmt/json.(*Encoder).flushValue(0x140002104d0?, 0x101ebddf9?)
        github.com/polydawn/refmt@v0.89.1-0.20221221234430-40501e09de1f/json/jsonEncoder.go:218 +0x204
github.com/polydawn/refmt/json.(*Encoder).Step(0x140002104d0, 0x14000562150)
        github.com/polydawn/refmt@v0.89.1-0.20221221234430-40501e09de1f/json/jsonEncoder.go:121 +0x89c
github.com/polydawn/refmt/shared.TokenPump.Run({{0x1019c9000?, 0x14000032960?}, {0x1019c9080?, 0x140002104d0?}})
        github.com/polydawn/refmt@v0.89.1-0.20221221234430-40501e09de1f/shared/pump.go:35 +0x80
main.cborToJson({0x14000afb824, 0x1da, 0x1da})
        github.com/bluesky-social/indigo/cmd/gosky/main.go:763 +0x33c
main.glob..func18.1({0x140000b2e00, 0x1b}, {{0x1400015d620?, 0x0?}})
        github.com/bluesky-social/indigo/cmd/gosky/main.go:640 +0x1b0
github.com/bluesky-social/indigo/repo.(*Repo).ForEach.func1({0x1, {0x140000b2e00, 0x1b}, {{0x1400015d620, 0x24}}, 0x0})
        github.com/bluesky-social/indigo/repo/repo.go:314 +0x4c
github.com/bluesky-social/indigo/mst.(*MerkleSearchTree).WalkLeavesFrom(0x14000133178?, {0x1019d2018, 0x140005886c0}, {0x0, 0x0}, 0x140001335d0)
        github.com/bluesky-social/indigo/mst/mst.go:936 +0x210
github.com/bluesky-social/indigo/mst.(*MerkleSearchTree).WalkLeavesFrom(0x140001332e8?, {0x1019d2018, 0x140005886c0}, {0x0, 0x0}, 0x140001335d0)
        github.com/bluesky-social/indigo/mst/mst.go:928 +0x110
github.com/bluesky-social/indigo/mst.(*MerkleSearchTree).WalkLeavesFrom(0x14000032230?, {0x1019d2018, 0x140005886c0}, {0x0, 0x0}, 0x140001335d0)
        github.com/bluesky-social/indigo/mst/mst.go:928 +0x110
github.com/bluesky-social/indigo/mst.(*MerkleSearchTree).WalkLeavesFrom(0x1400004c040?, {0x1019d2018, 0x140005886c0}, {0x0, 0x0}, 0x140001335d0)
        github.com/bluesky-social/indigo/mst/mst.go:928 +0x110
github.com/bluesky-social/indigo/repo.(*Repo).ForEach(0x140001c8000, {0x1019d1fa8, 0x140001a2008}, {0x0, 0x0}, 0x14000133700)
        github.com/bluesky-social/indigo/repo/repo.go:313 +0x148
main.glob..func18(0x14000527340)
        github.com/bluesky-social/indigo/cmd/gosky/main.go:623 +0x248
github.com/urfave/cli/v2.(*Command).Run(0x101f2d680, 0x14000527340, {0x14000527380, 0x4, 0x4})
        github.com/urfave/cli/v2@v2.25.1/command.go:274 +0x75c
github.com/urfave/cli/v2.(*Command).Run(0x140002286e0, 0x14000527140, {0x1400018e050, 0x5, 0x5})
        github.com/urfave/cli/v2@v2.25.1/command.go:267 +0x97c
github.com/urfave/cli/v2.(*App).RunContext(0x14000564000, {0x1019d1fa8?, 0x140001a2000}, {0x1400018e050, 0x5, 0x5})
        github.com/urfave/cli/v2@v2.25.1/app.go:332 +0x604
github.com/urfave/cli/v2.(*App).Run(...)
        github.com/urfave/cli/v2@v2.25.1/app.go:309
github.com/urfave/cli/v2.(*App).RunAndExitOnError(0x14000564000)
        github.com/urfave/cli/v2@v2.25.1/app.go:373 +0x48
main.run({0x1400006e768?, 0x100ec50e8?, 0x101ecd850?})
        github.com/bluesky-social/indigo/cmd/gosky/main.go:91 +0x5fc
main.main()
        github.com/bluesky-social/indigo/cmd/gosky/main.go:42 +0x30
exit status 2

[whyrusleeping]

oh yeah, this is a bug in the refmt library, it doesnt properly convert certain things. Need to just swap that out i think

[jmhodges]

I think this might be a generic problem with any CBOR to JSON conversion. For instance, CBOR allows for non-string keys in its maps and JSON libraries typically don't (because the JSON spec says not to). The panic in remft might be caused by that.

As evidence, I did a quick conversion from github.com/polydawn/refmt in cborToJson to github.com/fxamacker/cbor/v2 and encoding/json with a generic map[string]interface{} in the middle. That errored out because fxamacker/cbor returns the avatar and banner maps as map[interface{}][interface{}]. Those maps have only string keys, but a generic cbor parser can't assume that. Code in a branch on my fork

I believe there's a few options available:

1. Give up on the JSON encoding dream and print the raw Go types returned
2. Have an explicit set of types that can be pretty-printed and fallback to raw Go types or the original data when an unknown type is found (That is, parse the CBOR twice. Once to get the top-level $type value and the second to parse as the matching Go type for that value. It's unfortunate that the $type key is at the same level as the object's fields, but whaddaya gonna do)
3. Walk the generic maps to copy all of them and flip the key types to strings for maps and then error when the conversion is impossible. This assumes the cbor would never contain non-string keys for maps
4. Convince a cbor library like fxamacker/cbor to provide pretty-print tooling for generic data

[bnewbold]

FWIW, I'm in the middle of writing up better public specs for our data model (use of JSON and CBOR) and the Lexicon schema definition library. Those should end up in review over at https://github.com/bluesky-social/atproto-website/pulls in the next couple days, and might help nail down the space of data objects that need to be handled (or not).

[fxamacker]

That errored out because fxamacker/cbor returns the avatar and banner maps as map[interface{}][interface{}]. Those maps have only string keys, but a generic cbor parser can't assume that.

Hi @jmhodges, not sure if it can be used here, but the DefaultMapType decoding option in fxamacker/cbor can be useful when converting between CBOR and JSON. It specifies Go map type to create and decode to.

Instead of decoder returning "map[interface {}]interface {}", it will return "map[string]interface {}" like this:

b, _ := hex.DecodeString("a161616141") // {"a": "A"}

// Decoding option to allow user to specify custom default map type.
dm, _ := cbor.DecOptions{
	DefaultMapType: reflect.TypeOf(map[string]interface{}(nil)),
}.DecMode()

var v interface{}
_ = dm.Unmarshal(b, &v)

fmt.Printf("decoded: %v (%T)\n", v, v)

// decoded: map[a:A] (map[string]interface {})

https://go.dev/play/p/_og5ySCtqYN

Convince a cbor library like fxamacker/cbor to provide pretty-print tooling for generic data

BTW, it isn't exactly pretty printing, but recently fxamacker/cbor v2.5.0-beta3 added functions that produce Extended Diagnostic Notation which is defined in Appendix G of RFC 8610: Concise Data Definition Language (CDDL).

Hope this info is useful. FWIW, I found CDDL and EDN to be incredibly helpful in defining CBOR-based data format and creating a codec for it.

[jmhodges]

Well! The DefaultMapType config definitely fixed the original issue here! Pushed to my branch.

I'm not sure if there's other types where this would fail because I'm less familiar with the lexicon than the others in this thread. Seems like a plausible short term solution with a plausible long term follow-up of upgrading to cbor v2.5.0 when it's out.