Tests

[kirawi]

Resolves #6

All tests are present.

Note:
All AOTs, and all but one in-house tests fail. AOTs are failing because the output format is different than expected. This might be resolvable by simply cutting the gid prefixes from the output and reference output entirely. (done)

Previously disabled tests are also back, and so will need to be disabled again. This should probably be done through a master list of disabled tests in the generator. (done)

Correctly parsing variations and settings is also a possible TODO task, depending on whether they're necessary for the tests.

• Get all tests passing.
• Rebase and re-commit tests as --author="nobody <>".

Generator Tool

use std::{
  fmt::{self, Display},
  fs::File,
  io::{BufWriter, Write},
  mem,
  path::PathBuf,
};

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct ParseError;

impl fmt::Display for ParseError {
  fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
      write!(f, "missing closing quote")
  }
}

impl std::error::Error for ParseError {}

enum State {
  /// Within a delimiter.
  Delimiter,
  /// After backslash, but before starting word.
  Backslash,
  /// Within an unquoted word.
  Unquoted,
  /// After backslash in an unquoted word.
  UnquotedBackslash,
  /// Within a single quoted word.
  SingleQuoted,
  /// Within a double quoted word.
  DoubleQuoted,
  /// After backslash inside a double quoted word.
  DoubleQuotedBackslash,
  /// Inside a comment.
  Comment,
  InPath,
}

pub fn split(s: &str) -> Result<Vec<String>, ParseError> {
  use State::*;

  let mut words = Vec::new();
  let mut word = String::new();
  let mut chars = s.chars();
  let mut state = Delimiter;

  loop {
      let c = chars.next();
      state = match state {
          Delimiter => match c {
              None => break,
              Some('\'') => SingleQuoted,
              Some(c @ '[') => {
                  word.push(c);
                  SingleQuoted
              }
              Some('\"') => DoubleQuoted,
              Some('\\') => Backslash,
              Some('\t') | Some(' ') | Some('\n') | Some(':') | Some(';') => Delimiter,
              Some('#') => Comment,
              Some(c) => {
                  word.push(c);
                  Unquoted
              }
          },
          Backslash => match c {
              None => {
                  word.push('\\');
                  words.push(mem::replace(&mut word, String::new()));
                  break;
              }
              Some('\n') => Delimiter,
              Some(c) => {
                  word.push(c);
                  Unquoted
              }
          },
          Unquoted => match c {
              None => {
                  words.push(mem::replace(&mut word, String::new()));
                  break;
              }
              Some('\'') => SingleQuoted,
              Some(c @ '[') => {
                  word.push(c);
                  SingleQuoted
              }
              Some('\"') => DoubleQuoted,
              Some('\\') => UnquotedBackslash,
              Some('\t') | Some(' ') | Some('\n') | Some(':') | Some(';') => {
                  words.push(mem::replace(&mut word, String::new()));
                  Delimiter
              }
              Some(c @ '/') => {
                  word.push(c);
                  InPath
              }
              Some(c) => {
                  word.push(c);
                  Unquoted
              }
          },
          UnquotedBackslash => match c {
              None => {
                  word.push('\\');
                  words.push(mem::replace(&mut word, String::new()));
                  break;
              }
              Some('\n') => Unquoted,
              Some(c) => {
                  word.push(c);
                  Unquoted
              }
          },
          SingleQuoted => match c {
              None => return Err(ParseError),
              Some(c @ ']') => {
                  word.push(c);
                  Unquoted
              }
              Some('\'') => Unquoted,
              Some(c) => {
                  word.push(c);
                  SingleQuoted
              }
          },
          DoubleQuoted => match c {
              None => return Err(ParseError),
              Some('\"') => Unquoted,
              Some('\\') => DoubleQuotedBackslash,
              Some(c) => {
                  word.push(c);
                  DoubleQuoted
              }
          },
          DoubleQuotedBackslash => match c {
              None => return Err(ParseError),
              Some('\n') => DoubleQuoted,
              Some(c @ '$') | Some(c @ '`') | Some(c @ '"') | Some(c @ '\\') => {
                  word.push(c);
                  DoubleQuoted
              }
              Some(c) => {
                  word.push('\\');
                  word.push(c);
                  DoubleQuoted
              }
          },
          Comment => match c {
              None => break,
              Some('\n') => Delimiter,
              Some(_) => Comment,
          },
          InPath => match c {
              None => break,
              Some(';') => {
                  words.push(mem::replace(&mut word, String::new()));
                  Delimiter
              }
              Some(c) => {
                  word.push(c);
                  InPath
              }
          },
      }
  }

  Ok(words)
}

#[derive(Default, Debug)]
struct TestCase {
  name: String,
  path: PathBuf,
  font_size: usize,
  features: Vec<(String, u16)>,
  variations: Vec<(String, f32)>,
  input: String,
  output: Vec<String>,
  show_advance: bool,
  show_name: bool,
}

impl Display for TestCase {
  fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
      if self.path.starts_with("/") {
          write!(f, "#[cfg(target_os = \"macos\")]\n")?;
      }

      if self.output.contains(&"*".to_string()) {
          write!(
              f,
              "shaping_test!({}, {:?}, {}, &{:?}, &{:?}, {:?});",
              self.name,
              self.path.to_str().unwrap(),
              self.font_size,
              self.features,
              self.variations,
              self.input,
          )
      } else {
          write!(
              f,
              "shaping_test!({}, {:?}, {}, &{:?}, &{:?}, {:?}, &{:?}, {:?}, {:?});",
              self.name,
              self.path.to_str().unwrap(),
              self.font_size,
              self.features,
              self.variations,
              self.input,
              self.output,
              self.show_advance,
              self.show_name,
          )
      }
  }
}

impl TestCase {
  pub fn parse(
      arguments: Vec<String>,
      file_path: PathBuf,
      base_path: PathBuf,
      idx: usize,
  ) -> Self {
      let name = format!(
          "{}_{}",
          file_path
              .file_stem()
              .unwrap()
              .to_str()
              .unwrap()
              .replace("-", "_")
              .replace(" ", "_")
              .to_lowercase(),
          idx
      );

      let mut case = Self {
          name,
          show_advance: true,
          show_name: true,
          ..Self::default()
      };
      case.font_size = 75;

      for arg in arguments.iter() {
          if arg.starts_with("../") {
              // Path
              let arg = arg.strip_prefix("../fonts/").unwrap();
              case.path = base_path.join(arg);
          } else if arg.starts_with("/") {
              case.path = arg.into();
          } else if arg.starts_with("--") {
              // Features/Variations
              let arg = arg.strip_prefix("--").unwrap();
              if arg.starts_with("font-size=") {
                  let split: Vec<_> = arg.split("=").collect();
                  case.font_size = split.last().unwrap().parse().unwrap();
              } else if arg.contains("=") {
                  // Variation
                  let split: Vec<_> = arg.split("=").collect();
                  let variation: (String, f32) = (
                      split.first().unwrap().to_string(),
                      split.last().unwrap().parse().unwrap_or(0.0),
                  );
                  case.variations.push(variation);
              } else {
                  // Feature
                  match arg {
                      "no-positions" => case.show_advance = false,
                      "no-glyph-names" => case.show_name = false,
                      _ => case.features.push((arg.to_string(), 1)),
                  }
              }
          } else if arg.starts_with("[") || arg == "*" {
              // Output
              let output = arg.trim_matches(|chr| chr == '[' || chr == ']');
              for chr in output.split('|') {
                  case.output.push(chr.trim_start_matches("gid").to_string());
              }
          }
      }

      let input = arguments
          .get(
              arguments
                  .len()
                  .checked_sub(2)
                  .unwrap_or_else(|| panic!("{:#?}", arguments)),
          )
          .unwrap();
      for chr in input.split(',') {
          let chr = chr.trim_start_matches("U+");
          let parsed_chr: char =
              char::from_u32(u32::from_str_radix(chr, 16).expect(chr)).expect(chr);
          case.input.push(parsed_chr);
      }

      case
  }
}

fn main() -> std::io::Result<()> {
  let input = PathBuf::from("data");
  let output = PathBuf::from("output");
  std::fs::create_dir_all(output.clone())?;

  let shaping_tests = ["text-rendering-tests", "aots", "in-house"];
  let output_shaping_tests = ["text-rendering", "aot", "in-house"];
  let ignored_tests = vec![vec!["shlana", "cmap_3"], vec![], vec![]];
  for ((test, output_test), ignored_tests) in shaping_tests
      .iter()
      .zip(output_shaping_tests)
      .zip(ignored_tests)
  {
      let dir = input.join(test).join("tests");
      if dir.exists() {
          let entries = std::fs::read_dir(dir)?
              .flatten()
              .filter(|file| file.file_type().unwrap().is_file());
          let mut writer =
              BufWriter::new(File::create(output.join(format!("{}.rs", output_test)))?);
          writer.write("mod shaping;\n".as_bytes())?;

          // Iterate over each file in the directory and generate tests.
          for file in entries {
              if ignored_tests.iter().any(|x| {
                  file.file_name()
                      .to_string_lossy()
                      .to_lowercase()
                      .replace("-", "_")
                      .contains(x)
              }) {
                  continue;
              }
              let content = std::fs::read_to_string(file.path())?;
              for (idx, test) in content
                  .lines()
                  .map(|line| split(line).unwrap())
                  .filter(|line| !line.is_empty())
                  .enumerate()
              {
                  let formatted_string = format!(
                      "{}\n",
                      TestCase::parse(
                          test,
                          file.path(),
                          PathBuf::from("tests").join("fonts").join(output_test),
                          idx + 1
                      )
                  );
                  writer.write(formatted_string.as_bytes())?;
              }
          }
      }
  }

  Ok(())
}

[dfrg]

This looks fantastic. I've been buried in the layout crate for the last several weeks, but that work is effectively done for now. My focus at the moment is refactoring the Unicode/text analysis support into a separate crate. This work will also include improved complex cluster parsing which should address those failing Kannada tests and will also allow me to correct other Indic issues without ugly hacks. After that (in two weeks or so) shaping correctness will be my primary project so this will get my full attention.

I really appreciate all the work you've done on this.

[dfrg]

I have a pile of commits against this PR so I'm going to go ahead and merge it into a new branch so I can iterate on it while not disrupting CI on master.

Thanks again for the brilliant work on this!