Default namespaces cause maps to end parsing early

[cap10morgan]

Here's a test to reproduce:

fn test_default_map_namespace_syntax() {
    let input = "{:thingy #:foo{:bar \"baz\"} :more \"stuff\"}";
    let cfg = edn::read_string(&input).unwrap();
    if let Edn::Map(cfg_map) = cfg {
        assert!(cfg_map.contains_key(&Edn::Key("thingy")));
        assert!(cfg_map.contains_key(&Edn::Key("more")));        // test fails here
    }
}

Observed:

The test fails in the second assertion (i.e. that the map contains a :more key).

Expected:

Both assertions should pass.

If you remove the #:foo default map key namespace, both assertions pass.

[Grinkers]

Thanks for the first bug report! The core problem was I was assuming spaces between namespaces/tags. Interestingly, this wasn't caught by (cgen/ednable) fuzzer.

I haven't used namespaces/tags from clojure, so it would be greatly appreciated if you could verify this is the correct/expected behavior.

(modified and expanded version of your repro)

clojure-reader/tests/read.rs

Lines 240 to 283 in 67bb9cf

	#[test]
	fn test_default_map_namespace_syntax() {
	// see https://github.com/Grinkers/clojure-reader/issues/2
	let variations = [
	"{:thingy #:foo{:bar \"baz\"} :more \"stuff\"}",
	"{:thingy #:foo {:bar \"baz\"} :more \"stuff\"}",
	"{:more \"stuff\" :thingy #:foo{:bar \"baz\"}}",
	"{:more \"stuff\" :thingy # :foo{:bar \"baz\"}}",
	];
	for v in variations {
	let cfg = edn::read_string(&v).unwrap();
	let Edn::Map(cfg) = cfg else { panic!() };
	assert_eq!(
	cfg.get(&Edn::Key("thingy")),
	Some(&Edn::Tagged(
	":foo",
	Box::new(Edn::Map(BTreeMap::from([(Edn::Key("bar"), Edn::Str("baz"))])))
	))
	);
	assert_eq!(cfg.get(&Edn::Key("more")), Some(&Edn::Str("stuff")));
	}
	// without keyword `:` symbol.
	// the tag is parsed/preserved, but we don't support custom readers
	let variations = [
	"{:thingy #foo{:bar \"baz\"} :more \"stuff\"}",
	"{:thingy #foo {:bar \"baz\"} :more \"stuff\"}",
	"{:more \"stuff\" :thingy #foo{:bar \"baz\"}}",
	"{:more \"stuff\" :thingy # foo{:bar \"baz\"}}",
	];
	for v in variations {
	let cfg = edn::read_string(&v).unwrap();
	let Edn::Map(cfg) = cfg else { panic!() };
	assert_eq!(
	cfg.get(&Edn::Key("thingy")),
	Some(&Edn::Tagged(
	"foo",
	Box::new(Edn::Map(BTreeMap::from([(Edn::Key("bar"), Edn::Str("baz"))])))
	))
	);
	assert_eq!(cfg.get(&Edn::Key("more")), Some(&Edn::Str("stuff")));
	}

[cap10morgan]

@Grinkers Sorry for the delay. I'm hoping to take a look at this again today or very soon after. Thank you for your quick follow up.

[cap10morgan]

@Grinkers I'm not sure about the spaces, but the rules around this are defined here: https://clojure.org/reference/reader#_maps in the "Map namespace syntax" section if that's helpful.

[Grinkers]

@Grinkers I'm not sure about the spaces, but the rules around this are defined here: https://clojure.org/reference/reader#_maps in the "Map namespace syntax" section if that's helpful.

I believe this is for clojure the language/reader/interpreter.

https://github.com/edn-format/edn?tab=readme-ov-file#tagged-elements
For EDN, this is the relevant section. Technically the use case here is to be able to define custom handlers, which results in needing to fully implement clojure to handle all possible evaluations. I eventually hope to have a full lisp interpreter some day. I have some private experiments, but nothing ready to share.

I believe with this #3, everything will be parsed correctly with no loss of data. I assumed incorrectly that there was always going to be a space, but it's not required.

create a well-known generic representation that contains both the tag and the tagged element

is the current goal, leaving users able to do what they want with the parsed data.

[cap10morgan]

@Grinkers I'm not sure about the spaces, but the rules around this are defined here: https://clojure.org/reference/reader#_maps in the "Map namespace syntax" section if that's helpful.

I believe this is for clojure the language/reader/interpreter.

https://github.com/edn-format/edn?tab=readme-ov-file#tagged-elements
For EDN, this is the relevant section. Technically the use case here is to be able to define custom handlers, which results in needing to fully implement clojure to handle all possible evaluations. I eventually hope to have a full lisp interpreter some day. I have some private experiments, but nothing ready to share.

I believe with this #3, everything will be parsed correctly with no loss of data. I assumed incorrectly that there was always going to be a space, but it's not required.

create a well-known generic representation that contains both the tag and the tagged element

is the current goal, leaving users able to do what they want with the parsed data.

From that page defining EDN tagged elements:

edn supports extensibility through a simple mechanism. # followed immediately by a symbol starting with an alphabetic character indicates that that symbol is a tag.

The : at the start makes it a keyword, not a symbol; and it is not an alphabetic character.

This is a new-ish extension of map literal syntax, so it comes into EDN in the sense that EDN supports Clojure map literals.

Besides, your library is named clojure-reader ;)

[Grinkers]

One somewhat annoying thing about clojure and EDN is that there's an unofficial policy of being more accepting than the spec requires, so there's a lot of things that clojure will happily accept that is not specified anywhere.

Here's some examples of clojure (jvm) read-string:

;; the fix should be the rust equivalent of this
user=> (edn/read-string "{:thingy #:foo{:bar \"baz\"} :more \"stuff\"}")
{:thingy #:foo{:bar "baz"}, :more "stuff"}
(edn/read-string "{:more \"stuff\" :thingy #:foo {:bar \"baz\"}}")
{:more "stuff", :thingy #:foo{:bar "baz"}}

;; no support custom readers yet, but fix #3 should correctly read this so you can use the data
user=> (edn/read-string {:readers {'foo #(assoc % :read 42)}} "{:thingy #foo{:bar \"baz\"} :more \"stuff\"}")
{:thingy {:bar "baz", :read 42}, :more "stuff"}
user=> (edn/read-string {:readers {'foo #(assoc % :read 42)}} "{:thingy #foo {:bar \"baz\"} :more \"stuff\"}")
{:thingy {:bar "baz", :read 42}, :more "stuff"}

fwiw, I'm trying to be more accepting too, and matching edn/read-string as much as possible, even when the spec is ambiguous. As an example of what I do that edn/read-string does but is under-documented in the spec.

clojure-reader/tests/read.rs

Lines 100 to 116 in b00e51e

	#[test]
	fn parse_0x_ints() {
	assert_eq!(edn::read_string("0x2a").unwrap(), Edn::Int(42));
	assert_eq!(edn::read_string("-0X2A").unwrap(), Edn::Int(-42));
	// leading plus character
	assert_eq!(edn::read_string("+42").unwrap(), Edn::Int(42));
	assert_eq!(edn::read_string("+0x2a").unwrap(), Edn::Int(42));
	}
	#[test]
	fn parse_radix_ints() {
	assert_eq!(edn::read_string("16r2a").unwrap(), Edn::Int(42));
	assert_eq!(edn::read_string("8r63").unwrap(), Edn::Int(51));
	assert_eq!(edn::read_string("36rabcxyz").unwrap(), Edn::Int(623_741_435));
	assert_eq!(edn::read_string("-16r2a").unwrap(), Edn::Int(-42));
	assert_eq!(edn::read_string("-32rFOObar").unwrap(), Edn::Int(-529_280_347));
	}

I believe I now have tags correct, minus not supporting custom readers. Feedback/use case would be appreciated, because I haven't really used these features from clojure and things like "supply a custom reader at runtime" doesn't work so well with rust.

[Grinkers]

Merged and released as https://github.com/Grinkers/clojure-reader/releases/tag/0.4.0

Feel free to reopen if there's any issues.