Skip to content

Commit

Permalink
format,intepr: Refactor json, yaml, etc into formats also move out re…
Browse files Browse the repository at this point in the history 
…lated functions

json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).

Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
  • Loading branch information
@wader
wader committed Jul 23, 2022
1 parent 8b5cc89 commit cae288e
Show file tree
Hide file tree
Showing 95 changed files with 2,824 additions and 2,046 deletions.
1 change: 1 addition & 0 deletions .gitattributes
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@
*.fqtest eol=lf
*.json eol=lf
*.jq eol=lf
*.xml eol=lf
6 changes: 6 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ bsd_loopback_frame,
[bson](doc/formats.md#bson),
bzip2,
[cbor](doc/formats.md#cbor),
[csv](doc/formats.md#csv),
dns,
dns_tcp,
elf,
Expand All @@ -82,6 +83,7 @@ hevc_nalu,
hevc_pps,
hevc_sps,
hevc_vps,
[html](doc/formats.md#html),
icc_profile,
icmp,
icmpv6,
Expand Down Expand Up @@ -120,6 +122,7 @@ sll_packet,
tar,
tcp_segment,
tiff,
toml,
udp_datagram,
vorbis_comment,
vorbis_packet,
Expand All @@ -130,6 +133,8 @@ vpx_ccr,
wav,
webp,
xing,
[xml](doc/formats.md#xml),
yaml,
[zip](doc/formats.md#zip)

[#]: sh-end
Expand Down Expand Up @@ -286,3 +291,4 @@ Licenses of direct dependencies:
- golang/snappy https://github.com/golang/snappy/blob/master/LICENSE (BSD)
- github.com/BurntSushi/toml https://github.com/BurntSushi/toml/blob/master/COPYING (MIT)
- gopkg.in/yaml.v3 https://github.com/go-yaml/yaml/blob/v3/LICENSE (MIT)
- github.com/creasty/defaults https://github.com/creasty/defaults/blob/master/LICENSE (MIT)
72 changes: 70 additions & 2 deletions doc/formats.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
|[`bson`](#bson) |Binary&nbsp;JSON |<sub></sub>|
|`bzip2` |bzip2&nbsp;compression |<sub>`probe`</sub>|
|[`cbor`](#cbor) |Concise&nbsp;Binary&nbsp;Object&nbsp;Representation |<sub></sub>|
|[`csv`](#csv) |Comma&nbsp;separated&nbsp;values |<sub></sub>|
|`dns` |DNS&nbsp;packet |<sub></sub>|
|`dns_tcp` |DNS&nbsp;packet&nbsp;(TCP) |<sub></sub>|
|`elf` |Executable&nbsp;and&nbsp;Linkable&nbsp;Format |<sub></sub>|
Expand All @@ -52,6 +53,7 @@
|`hevc_pps` |H.265/HEVC&nbsp;Picture&nbsp;Parameter&nbsp;Set |<sub></sub>|
|`hevc_sps` |H.265/HEVC&nbsp;Sequence&nbsp;Parameter&nbsp;Set |<sub></sub>|
|`hevc_vps` |H.265/HEVC&nbsp;Video&nbsp;Parameter&nbsp;Set |<sub></sub>|
|[`html`](#html) |HyperText&nbsp;Markup&nbsp;Language |<sub></sub>|
|`icc_profile` |International&nbsp;Color&nbsp;Consortium&nbsp;profile |<sub></sub>|
|`icmp` |Internet&nbsp;Control&nbsp;Message&nbsp;Protocol |<sub></sub>|
|`icmpv6` |Internet&nbsp;Control&nbsp;Message&nbsp;Protocol&nbsp;v6 |<sub></sub>|
Expand All @@ -61,7 +63,7 @@
|`ipv4_packet` |Internet&nbsp;protocol&nbsp;v4&nbsp;packet |<sub>`ip_packet`</sub>|
|`ipv6_packet` |Internet&nbsp;protocol&nbsp;v6&nbsp;packet |<sub>`ip_packet`</sub>|
|`jpeg` |Joint&nbsp;Photographic&nbsp;Experts&nbsp;Group&nbsp;file |<sub>`exif` `icc_profile`</sub>|
|`json` |JSON |<sub></sub>|
|`json` |JavaScript&nbsp;Object&nbsp;Notation |<sub></sub>|
|[`macho`](#macho) |Mach-O&nbsp;macOS&nbsp;executable |<sub></sub>|
|[`matroska`](#matroska) |Matroska&nbsp;file |<sub>`aac_frame` `av1_ccr` `av1_frame` `avc_au` `avc_dcr` `flac_frame` `flac_metadatablocks` `hevc_au` `hevc_dcr` `image` `mp3_frame` `mpeg_asc` `mpeg_pes_packet` `mpeg_spu` `opus_packet` `vorbis_packet` `vp8_frame` `vp9_cfm` `vp9_frame`</sub>|
|[`mp3`](#mp3) |MP3&nbsp;file |<sub>`id3v2` `id3v1` `id3v11` `apev2` `mp3_frame`</sub>|
Expand Down Expand Up @@ -90,6 +92,7 @@
|`tar` |Tar&nbsp;archive |<sub>`probe`</sub>|
|`tcp_segment` |Transmission&nbsp;control&nbsp;protocol&nbsp;segment |<sub></sub>|
|`tiff` |Tag&nbsp;Image&nbsp;File&nbsp;Format |<sub>`icc_profile`</sub>|
|`toml` |Tom's&nbsp;Obvious,&nbsp;Minimal&nbsp;Language |<sub></sub>|
|`udp_datagram` |User&nbsp;datagram&nbsp;protocol |<sub>`udp_payload`</sub>|
|`vorbis_comment` |Vorbis&nbsp;comment |<sub>`flac_picture`</sub>|
|`vorbis_packet` |Vorbis&nbsp;packet |<sub>`vorbis_comment`</sub>|
Expand All @@ -100,12 +103,14 @@
|`wav` |WAV&nbsp;file |<sub>`id3v2` `id3v1` `id3v11`</sub>|
|`webp` |WebP&nbsp;image |<sub>`vp8_frame`</sub>|
|`xing` |Xing&nbsp;header |<sub></sub>|
|[`xml`](#xml) |Extensible&nbsp;Markup&nbsp;Language |<sub></sub>|
|`yaml` |YAML&nbsp;Ain't&nbsp;Markup&nbsp;Language |<sub></sub>|
|[`zip`](#zip) |ZIP&nbsp;archive |<sub>`probe`</sub>|
|`image` |Group |<sub>`gif` `jpeg` `mp4` `png` `tiff` `webp`</sub>|
|`inet_packet` |Group |<sub>`ipv4_packet` `ipv6_packet`</sub>|
|`ip_packet` |Group |<sub>`icmp` `icmpv6` `tcp_segment` `udp_datagram`</sub>|
|`link_frame` |Group |<sub>`bsd_loopback_frame` `ether8023_frame` `sll2_packet` `sll_packet`</sub>|
|`probe` |Group |<sub>`adts` `ar` `avro_ocf` `bitcoin_blkdat` `bzip2` `elf` `flac` `gif` `gzip` `jpeg` `json` `macho` `matroska` `mp3` `mp4` `mpeg_ts` `ogg` `pcap` `pcapng` `png` `tar` `tiff` `wav` `webp` `zip`</sub>|
|`probe` |Group |<sub>`adts` `ar` `avro_ocf` `bitcoin_blkdat` `bzip2` `elf` `flac` `gif` `gzip` `jpeg` `json` `macho` `matroska` `mp3` `mp4` `mpeg_ts` `ogg` `pcap` `pcapng` `png` `tar` `tiff` `toml` `wav` `webp` `xml` `yaml` `zip`</sub>|
|`tcp_stream` |Group |<sub>`dns` `rtmp`</sub>|
|`udp_payload` |Group |<sub>`dns`</sub>|

Expand Down Expand Up @@ -280,6 +285,27 @@ Supports `torepr`
- https://en.wikipedia.org/wiki/CBOR
- https://www.rfc-editor.org/rfc/rfc8949.html

### csv

#### Options

|Name |Default|Description|
|- |- |-|
|`comma` |, |Separator character|
|`comment`|# |Comment line character|

#### Examples

Decode file using csv options
```
$ fq -d csv -o comma="," -o comment="#" . file
```

Decode value as csv
```
... | csv({comma:",",comment:"#"})
```

### flac_frame

#### Options
Expand Down Expand Up @@ -320,6 +346,27 @@ Decode value as hevc_au
... | hevc_au({length_size:4})
```

### html

#### Options

|Name |Default|Description|
|- |- |-|
|`array`|false |Decode as nested arrays|
|`seq` |false |Use seq attribute to preserve element order|

#### Examples

Decode file using html options
```
$ fq -d html -o array=false -o seq=false . file
```

Decode value as html
```
... | html({array:false,seq:false})
```

### macho

Supports decoding vanilla and FAT Mach-O binaries.
Expand Down Expand Up @@ -456,6 +503,27 @@ Current only supports plain RTMP (not RTMPT or encrypted variants etc) with AMF0
- https://rtmp.veriskope.com/docs/spec/
- https://rtmp.veriskope.com/pdf/video_file_format_spec_v10.pdf

### xml

#### Options

|Name |Default|Description|
|- |- |-|
|`array`|false |Decode as nested arrays|
|`seq` |false |Use seq attribute to preserve element order|

#### Examples

Decode file using xml options
```
$ fq -d xml -o array=false -o seq=false . file
```

Decode value as xml
```
... | xml({array:false,seq:false})
```

### zip

Supports ZIP64.
Expand Down

0 comments on commit cae288e

Please sign in to comment.