Feature idea: support ASN.1 (DER/BER)

[gjvnq]

Hey, it would be pretty cool with fq supported decoding binary¹ ASN.1 (Abstract Syntax Notation One) as it is used in various communication protocols.

¹ ASN.1 has many encoding options but pretty much everything that is binary encoded seems to be either BER or DER.

The main problem I see is that the field names are not included in the actual files, so either the user would have to point to an ASN.1 grammar/spec file or the output would be "nameless".

[wader]

Hi, yes that is a good idea. It would also require figuring out a nice way of passing in a grammar file to the decoder.
If you want to work on this i would recommend having a look at the protobutf decoder https://github.com/wader/fq/blob/master/format/protobuf/protobuf.go. It does not support passing in schemas but other decoders can pass it something schema-like internally to improve the decode output.

[gjvnq]

Thanks. I will take a look if I have time in January.

[wader]

👍 There are some other issues #25 #24 that will share som issues with this like how to provide an external description file and how to model the decoded value tree.

[wader]

Hey, some update. I have a basic BER decoder working now, also working on adding support for arbitrary large integer to the decoder which will be useful for ASN.1. and make it more useful for certificates and crypto related things i think. Hope i can push something in the next few days.

[wader]

Hey, now there is basic ASN1 BER support in master and also some kind of torepr support.

To make it a bit easer to work with PEM there is also frompem/topem now.

Any feedback what could be done differently or bugs are much appreciated.

Example reading PEM file, decode it and display it as decode tree, JSON and torepr JSON:

$ fq -o line_bytes=10 -d raw '(. | tobytes | tostring | println), (frompem | asn1_ber | d, tovalue, torepr)' format/asn1/testdata/test.pem
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDMYfnvWtC8Id5bPKae5yXSxQTt
+Zpul6AnnZWfI2TtIarvjHBFUtXRo96y7hoL4VWOPKGCsRqMFDkrbeUjRrx8iL91
4/srnyf6sh9c8Zk04xEOpK1ypvBz+Ks4uZObtjnnitf0NBGdjMKxveTq+VE7BWUI
yQjtQ8mbDOsiLLvh7wIDAQAB
-----END PUBLIC KEY-----
    │00 01 02 03 04 05 06 07 08 09│0123456789│.{}: (asn1_ber)
0x00│30                           │0         │  class: "universal" (0)
0x00│30                           │0         │  form: "constructed" (1)
0x00│30                           │0         │  tag: "sequence" (0x10)
0x00│   81 9f                     │ ..       │  length: 159
    │                             │          │  constructed[0:2]:
    │                             │          │    [0]{}:
0x00│         30                  │   0      │      class: "universal" (0)
0x00│         30                  │   0      │      form: "constructed" (1)
0x00│         30                  │   0      │      tag: "sequence" (0x10)
0x00│            0d               │    .     │      length: 13
    │                             │          │      constructed[0:2]:
    │                             │          │        [0]{}:
0x00│               06            │     .    │          class: "universal" (0)
0x00│               06            │     .    │          form: "primitive" (0)
0x00│               06            │     .    │          tag: "object_identifier" (0x6)
0x00│                  09         │      .   │          length: 9
    │                             │          │          value[0:7]:
0x00│                     2a      │       *  │            [0]: 1
0x00│                     2a      │       *  │            [1]: 2
0x00│                        86 48│        .H│            [2]: 840
0x0a│86 f7 0d                     │...       │            [3]: 113549
0x0a│         01                  │   .      │            [4]: 1
0x0a│            01               │    .     │            [5]: 1
0x0a│               01            │     .    │            [6]: 1
    │                             │          │        [1]{}:
0x0a│                  05         │      .   │          class: "universal" (0)
0x0a│                  05         │      .   │          form: "primitive" (0)
0x0a│                  05         │      .   │          tag: "null" (0x5)
0x0a│                     00      │       .  │          length: "indefinite" (0)
    │                             │          │          value: null
    │                             │          │    [1]{}:
0x0a│                        03   │        . │      class: "universal" (0)
0x0a│                        03   │        . │      form: "primitive" (0)
0x0a│                        03   │        . │      tag: "bit_string" (0x3)
0x0a│                           81│         .│      length: 141
0x14│8d                           │.         │
0x14│   00                        │ .        │      unused_bits_count: 0
0x14│      30 81 89 02 81 81 00 cc│  0.......│      value: raw bits
0x1e│61 f9 ef 5a d0 bc 21 de 5b 3c│a..Z..!.[<│
0x28│a6 9e e7 25 d2 c5 04 ed f9 9a│...%......│
*   │until 0xa1.7 (end) (140)     │          │
{
  "class": "universal",
  "constructed": [
    {
      "class": "universal",
      "constructed": [
        {
          "class": "universal",
          "form": "primitive",
          "length": 9,
          "tag": "object_identifier",
          "value": [
            1,
            2,
            840,
            113549,
            1,
            1,
            1
          ]
        },
        {
          "class": "universal",
          "form": "primitive",
          "length": "indefinite",
          "tag": "null",
          "value": null
        }
      ],
      "form": "constructed",
      "length": 13,
      "tag": "sequence"
    },
    {
      "class": "universal",
      "form": "primitive",
      "length": 141,
      "tag": "bit_string",
      "unused_bits_count": 0,
      "value": "<140>MIGJAoGBAMxh+e9a0Lwh3ls8pp7nJdLFBO35mm6XoCedlZ8jZO0hqu+McEVS1dGj3rLuGgvhVY48oYKxGowUOStt5SNGvHyIv3Xj+yufJ/qyH1zxmTTjEQ6krXKm8HP4qzi5k5u2OeeK1/Q0EZ2MwrG95Or5UTsFZQjJCO1DyZsM6yIsu+HvAgMBAAE="
    }
  ],
  "form": "constructed",
  "length": 159,
  "tag": "sequence"
}
[
  [
    [
      1,
      2,
      840,
      113549,
      1,
      1,
      1
    ],
    null
  ],
  "<0b10001100>MIGJAoGBAMxh+e9a0Lwh3ls8pp7nJdLFBO35mm6XoCedlZ8jZO0hqu+McEVS1dGj3rLuGgvhVY48oYKxGowUOStt5SNGvHyIv3Xj+yufJ/qyH1zxmTTjEQ6krXKm8HP4qzi5k5u2OeeK1/Q0EZ2MwrG95Or5UTsFZQjJCO1DyZsM6yIsu+HvAgMBAAE="
]