Example parsers

[Geal]

We currently have a few example parsers. In order to test the project and make it useful, other formats can be implemented. Here is a list, if anyone wants to try it:

• text file formats:
  • INI
  • FASTQ
  • libconfig-like configuration file format
  • torrc configuration file
  • ISO 8601 dates
  • Web archive
  • TOML
  • bencode
  • CSV
  • YAML
  • CommonMark
• audio, video and image formats:
  • MP4 (partial implementation)
  • GIF
  • FLAC
  • FLV
  • OGG
  • MKV
  • MPEG TS
  • AVI
  • PNG
  • JPEG
  • EXIF
  • MP3
• document formats:
  • torrent files
  • TAR
  • PDF
  • MS-CFB (compound format, used in doc, xls, ppt, cab, msi files)
  • GZ
  • ZIP
  • RAR
  • binary PLIST
• database formats:
  • Redis database files
  • Ceph crush maps
• network protocol formats:
  • IRC
  • Pcap-NG
  • IP
  • Ethernet
  • TCP
  • UDP
  • NTP
  • DNS
  • PCAP
• executable formats:
  • Portable executables (PE)
  • ELF
  • GameBoy ROM
• crypto related:
  • ASN.1
  • X.509 certificates
  • DER public and private keys
  • SSL/TLS packets
  • OpenPGP
• Programming Languages
  • Rust
  • Lua
  • Python
  • C
• interface definition formats:
  • Thrift
  • Protobuf
  • AIDL

[thehydroimpulse]

I'm writing a Thrift library for Rust that'll use Nom for both their IDL and the network protocol, so that can be another example (although in a different repo).

[Geal]

Nice idea, that will be useful! Please notify me when it is done, I will add a link in this list.

[filipegoncalves]

This looks interesting. Is anyone actively working on any of these parsers? I'd like to work on a few of these.

[Geal]

I have some code for a GIF one at https://github.com/Geal/gif.rs but it is hard to test, since the graphical tools in Piston change a lot.

You can pick any of them. Network packets may be the easiest, since they don't require a decompression phase.

I am using the gif example to see what kind of API can be built over nom. Most of the parsing example are done as one pass over the data, but often there is some logic on the side, and it is not easy to encode correctly.

[elij]

I've started a fastq (http://en.wikipedia.org/wiki/FASTQ_format) parser https://github.com/elij/fastq.rs

[Geal]

@elij this is a great idea! Was it easy to do?

[elij]

yup it's a great framework -- though I struggled a bit with eof so I borrowed some code from rust-config (https://github.com/elij/fastq.rs/blob/master/src/parser.rs#L69) -- is there a better solution?

[Geal]

yes, eof should be a parser provided by nom, I am just waiting for @filipegoncalves to send a PR 😉

[filipegoncalves]

Hah, sorry for my silence. I've been busy lately. I just sent a PR (#31).

I will be working on one of these example parsers as soon as I get some spare time. There are some great ideas in here!

[Keruspe]

I might give tar a try

[nelsonjchen]

Does this check off PCAP?

https://github.com/richo/pcapng-rs

[Geal]

pcap-ng and pcap are two different formats, right? It seems the consensus now is to move everything to pcap-ng, though.

[TechnoMancer]

I will try a FLAC parser, need to add quite a few things for it though.

[badboy]

ISO8601 is done in https://github.com/badboy/iso8601 (I hope it's mostly correct.)

[Geal]

ok, it should be up to date. More to come 😄

[sbeckeriv]

WARC file format released. https://crates.io/crates/warc_parser

[Geal]

@sbeckeriv great, thanks!

[porglezomp]

It might be informative to try parsing the rust grammar with nom, if nobody has yet. In any case, I'd like to see a few programming languages on that list, since that's my use case.

[Geal]

@porglezomp programming languages examples would definitely be useful, but the Rust grammar might be a bit too much for the first attempt. Which other languages would you like to handle?

[porglezomp]

Yeah, I'm aware of the scale problem of Rust. I don't want to write that one, but I think it's a good holy grail for any parser library written in Rust. I'd like to try parsing the Lua grammar first, I think.

I recommend adding to the list:

• Programming Languages
  • Rust
  • Lua (I'll do this)
  • Python (or some other whitespace significant language)
  • C

[Geal]

ok, I added them to the list :)

[chriskrycho]

You have INI marked as done; do you have a link to it? (I'd love to use this for some tooling I'm hoping to build in 2016; need a good non-trivial example for it, though.)

[badboy]

@chriskrycho: https://github.com/Geal/nom/blob/master/tests/ini.rs

[chriskrycho]

Thanks very much, @badboy!

[fbernier]

I'll try to make the TOML parser very soon.

[Geal]

Actually, I think I should rewrite that INI parser, now that more convenient combinators are available.
Also, I should really work on that combinator for space separated stuff

[Geal]

@fbernier great! Please keep me posted!

[l0calh05t]

Maybe add a simple example for trailing commas in lists? Python has those, but is quite complex. Can't think of a simple example though.

[johshoff]

That IRC example is no longer using nom. The parser was moved into its own repository: https://github.com/Detegr/RBot-parser

[dtolnay]

Yeah, I'm aware of the scale problem of Rust. I don't want to write that one, but I think it's a good holy grail for any parser library written in Rust.

As of version 0.10.0, syn is now able to parse practically all of Rust syntax. One of my test cases is to parse the entire github.com/rust-lang/rust repo into an AST and print it back out, asserting that the output is identical to the original.

I am technically not using nom but instead a fork which removes the IResult::Incomplete variant. I found that the extra macro code generated to handle Incomplete was more than doubling the compile time for something that I didn't even want. Nevertheless, the code is enough like nom that I think we can check off the box.

Example snippet to parse one arm of a match expression:

named!(match_arm -> Arm, do_parse!(
    attrs: many0!(outer_attr) >>
    pats: separated_nonempty_list!(punct!("|"), pat) >>
    guard: option!(preceded!(keyword!("if"), expr)) >>
    punct!("=>") >>
    body: alt!(
        map!(block, |blk| ExprKind::Block(BlockCheckMode::Default, blk).into())
        |
        expr
    ) >>
    (Arm {
        attrs: attrs,
        pats: pats,
        guard: guard.map(Box::new),
        body: Box::new(body),
    })
));

[Geal]

@dtolnay syn is an amazing example, thanks for your hard work :)

[Geal]

@dtolnay could I get your input on #356? It might fix your issues with compile times, so I'd like to get your thoughts on this.

[J-F-Liu]

I am writing a PDF library using nom to parse PDF syntax. Released v0.1.0 just now.
https://github.com/J-F-Liu/lopdf

[valarauca]

So I've implemented a EDI parser for the ANS standard EDI for work with this. Awesome library really useful. Sadly that's owned by my employer.

I've started implementing an x64 assembler with nom. I'm really struggling with writing the parser. The main reason is register names have a lot of overlap, and are very short. For example r8, r8w, r11, and r12d. Ideally I want to map these to an enum. map!() makes this easy, but how can I match those terms in nom?

[Keruspe]

I converted several "keys" to enum values in my brainfuck parser, might or might not be relevant to your needs. See the first parsers defined with "named!" https://github.com/Keruspe/brainfuck.rs/blob/master/src/parser.rs

[przygienda]

is there a way (or it would be great if it's possible) to generate EBNF from this? Great package BTW ...

[ithinuel]

Hi,
I just pushed a pcap parser : https://github.com/ithinuel/pcap-rs.
It still needs the PR #492 to be merge so it can use official nom crate.

Any feedback is welcome.

[bbqsrc]

A parser for the Mediawiki format would be quite useful.

[dwerner]

@Geal thanks for an awesome library! I wrote a wavefront obj/mtl 3d mesh parser using it nom-obj, which I published to crates.io

[olivren]

I wrote a parser for the simple key/value text format .properties, which is a standard for Java configuration files. It uses nom 3.1. Can it be added to the list?

This is the first parser I wrote using a Parser Combinator library. If anyone can review my code I would be delighted. Also, I tried to add error reporting to my code, but I gave up after I tried to insert add_return_error and return_error calls all over the place to no avail (in the branch "error-reporting"). Is there an example of a text parser that reports parsing errors?

Edit: I rewrote my library using Pest instead of Nom, as I find it more suited to parsing a text format. I will definitely use nom if I need to parse a binary format, though.

[santifa]

@Geal thanks for this library.
I've implemented a parser for URI's which is
part of a larger side project for RDF (n3, ttl,...) parsers. The full abnf of rfc 3986 is implemented but the pct-encoding is still a bit messy.

[dbrgn]

Here's a parser for ICE candidates SDP (RFC 5245), used for example in WebRTC: https://github.com/dbrgn/candidateparser

[kamarkiewicz]

I wrote a Session Initiation Protocol (RFC3261) low-level push parser with API inspired by seanmonstar/httparse (hyper's HTTP parser):
https://github.com/kamarkiewicz/parsip

[thejpster]

I'd be interested in something that could parse SNMP MIB and YANG.

https://en.wikipedia.org/wiki/YANG

[ctrlcctrlv]

The BitTorrent example has been deleted, it seems.

[Riduidel]

As a beginner in Rust world, I'm quite sure I will say something horribly wrong, but is there any planned support for some XML dialects ? (typically RSS/ATOM) ?

[dwerner]

Nothing at all wrong with asking, and I'm sure someone might want to implement one at some point, but this is a list of example parsers written using nom, rather than a list of formats "supported" by nom. An xml parser would be an excellent idea for learning nom, imo.

[porglezomp]

@Riduidel if you're specifically interested in just having parsers for those formats, look at https://github.com/rust-syndication. I don't think there's any nom involved there though.

[mitchtbaum]

HTTP: https://github.com/hjr3/weldr/blob/00481f80ae60bd6b312805245c126c168ab77b36/src/http/parser.rs

[vandenoever]

A parser for Turtle. It passes the test suite in 15ms.

https://github.com/vandenoever/rome/tree/master/src/io/turtle

[ProgVal]

I wrote a Python parser: https://docs.rs/python-parser/

[idursun]

I think Redis database file format parser is not using nom at all. I couldn't find any reference to nom anywhere.

[nelsonjchen]

@idursun Maybe it refers to this old branch from a year before the last update to master. https://github.com/badboy/rdb-rs/tree/nom-parser