[BUG] Formats with digit_separators can't parse float numbers

[Robert42]

Description

Trying to define a custom format (a format containing digit separators), I couldn't get my number format to parse the string "42.0". After a while I've noticed, that those provided formats, which also contain digit separators, can't parse the same string either. See test case.

Prerequisites

• Rust version: rustc 1.54.0 (a178d0322 2021-07-26)
• lexical version: 6.0.0
• lexical compilation features used: format

Test case

fn main() {
  const RUST: u128 = lexical::format::RUST_LITERAL;
  const JSON: u128 = lexical::format::JSON;
  const CXX: u128 = lexical::format::CXX17_LITERAL;

  let o = lexical::ParseFloatOptions::new();
  
  println!("{:?}", lexical::parse_with_options::<f64, _, JSON>("42.0", &o));
  
  // RUST_LITERAL
  println!("{:?}", lexical::parse_with_options::<f64, _, RUST>("42.0", &o));
  println!("{:?}", lexical::parse_with_options::<f64, _, RUST>("4_2.0", &o));
  
  // CXX17_LITERAL
  println!("{:?}", lexical::parse_with_options::<f64, _, CXX>("42.0", &o));
  println!("{:?}", lexical::parse_with_options::<f64, _, CXX>("4'2.0", &o));
}

I would expect all five println invocations to print Ok(42.0). But in the actual output, only the first one is able to parse the number.

Ok(42.0)
Err(EmptyInteger(2))
Err(EmptyInteger(3))
Err(EmptyMantissa(4))
Err(EmptyMantissa(5))

Additional Context

When I copy the RUST_LITERAL and CXX17_LITERAL definitions to my main function and comment out the digit_separator, the simple case can be parsed correctly:

  pub const CXX_NOSEP: u128 = lexical::NumberFormatBuilder::new()
//    .digit_separator(std::num::NonZeroU8::new(b'\''))
    .case_sensitive_special(true)
    .internal_digit_separator(true)
    .build();
  println!("{:?}", lexical::parse_with_options::<f64, _, CXX_NOSEP>("42.0", &o));

  pub const RUST_NO_SEP: u128 = lexical::NumberFormatBuilder::new()
//    .digit_separator(std::num::NonZeroU8::new(b'_'))
    .required_digits(true)
    .no_positive_mantissa_sign(true)
    .no_special(true)
    .internal_digit_separator(true)
    .trailing_digit_separator(true)
    .consecutive_digit_separator(true)
    .build();
  println!("{:?}", lexical::parse_with_options::<f64, _, RUST_NO_SEP>("42.0", &o));

prints

Ok(42.0)
Ok(42.0)

[Alexhuszagh]

Closed as of lexical-util v0.8.1. Remember to run cargo update on any crates using lexical and it should work.

A working sample is as follows:

Cargo.toml

[package]
name = "lexical_test"
version = "0.1.0"
edition = "2018"

[dependencies.lexical]
version = "6.0"
features=["format"]

main.rs

fn main() {
  const RUST: u128 = lexical::format::RUST_LITERAL;
  const JSON: u128 = lexical::format::JSON;
  const CXX: u128 = lexical::format::CXX17_LITERAL;

  let o = lexical::ParseFloatOptions::new();

  println!("{:?}", lexical::parse_with_options::<f64, _, JSON>("42.0", &o));

  // RUST_LITERAL
  println!("{:?}", lexical::parse_with_options::<f64, _, RUST>("42.0", &o));
  println!("{:?}", lexical::parse_with_options::<f64, _, RUST>("4_2.0", &o));

  // CXX17_LITERAL
  println!("{:?}", lexical::parse_with_options::<f64, _, CXX>("42.0", &o));
  println!("{:?}", lexical::parse_with_options::<f64, _, CXX>("4'2.0", &o));
}

Tests for regressions have been added here.

[Robert42]

Wow, that was fast! Thank you.

The issue is now also gone in my actual project!

[Alexhuszagh]

Wow, that was fast! Thank you.

The issue is now also gone in my actual project!

Perfect, glad to hear it. Luckily, because of the re-structuring of the digit separator logic, this fix only required patching 3 lines of code, so once I saw this issue, it was quite an easy patch.

Let me know if there are any further issues, I've tried to extensively test functionality but sometimes bugs fall through the cracks.