Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with compiled format #1324

Closed
dkavolis opened this issue Sep 23, 2019 · 6 comments
Closed

Issues with compiled format #1324

dkavolis opened this issue Sep 23, 2019 · 6 comments

Comments

@dkavolis
Copy link
Contributor

I've been trying out compiling format strings and run into a few issues when using custom types with examples at the end (I know it's still experimental):

  • get function fails to compile when using multiple type arguments in fmt::compile

    fmt/include/fmt/compile.h

    Lines 332 to 339 in 758446c

    // Returns a reference to the argument at index N from [first, rest...].
    template <int N, typename T, typename... Args>
    const auto& get(const T& first, const Args&... rest) {
    if constexpr (sizeof...(Args) == 0)
    return first;
    else
    return get(rest...);
    }

It should look like:

// Returns a reference to the argument at index N from [first, rest...].
template <int N, typename T, typename... Args>
const auto& get(const T& first, const Args&... rest) {
  static_assert(N < 1 + sizeof...(Args), "");
  if constexpr (N == 0)
    return first;
  else
    return get<N - 1>(rest...);
}
constexpr auto f1 = fmt::compile<double>(FMT_STRING("{}\n"));
std::cout << fmt::format(f1, formattable{2});

Maybe the compiled format should contain template arguments that could later be used to enforce type safety at compile time but still allowing implicit conversions?

  • Format specifiers don't work with custom types:
constexpr auto f2 = fmt::compile<formattable>(FMT_STRING("{:05}\n"));
std::cout << fmt::format(f2, formattable{2});

crashes at runtime with

terminate called after throwing an instance of 'fmt::v6::format_error'
  what():  format specifier requires numeric argument

Stack:

__GI_raise 0x00007fd078350ed7
__GI_abort 0x00007fd078332535
<unknown> 0x00007fd0785ab672
<unknown> 0x00007fd0785b72b6
std::terminate() 0x00007fd0785b7301
__cxa_throw 0x00007fd0785b7535
fmt::v6::internal::error_handler::on_error(char const*) [clone .cold.573] 0x0000555efd7d2400
fmt::v6::internal::numeric_specs_checker<fmt::v6::internal::error_handler>::require_numeric_argument format.h:2020
fmt::v6::internal::cf::vformat_to<fmt::v6::basic_format_context<std::back_insert_iterator<fmt::v6::internal::buffer<char> >, char>, fmt::v6::buffer_range<char>, const fmt::v6::internal::compiled_format<main()::<lambda()>::str, formattable> >(fmt::v6::buffer_range<char>, const fmt::v6::internal::compiled_format<main()::<lambda()>::str, formattable> &, fmt::v6::basic_format_args<fmt::v6::basic_format_context<std::back_insert_iterator<fmt::v6::internal::buffer<char> >, char> >) compile.h:219
fmt::v6::format<fmt::v6::internal::compiled_format<main()::<lambda()>::str, formattable>, formattable>(const fmt::v6::internal::compiled_format<main()::<lambda()>::str, formattable> &, const formattable &) compile.h:542
main main.cpp:52
__libc_start_main 0x00007fd078333b6b
_start 0x0000555efd7d25ba
  • Format parsing seems to be ignored:
    std::cout << fmt::format(FMT_STRING("{:!}\n"), 5.3); fails to compile while
    constexpr auto f4 = fmt::compile<double>(FMT_STRING("{:!}\n")); compiles fine.
    std::cout << fmt::format(f4, 42.0); crashes at runtime with:
terminate called after throwing an instance of 'fmt::v6::format_error'
  what():  invalid type specifier

Stack:

__GI_raise 0x00007f303e781ed7
__GI_abort 0x00007f303e763535
<unknown> 0x00007f303e9dc672
<unknown> 0x00007f303e9e82b6
std::terminate() 0x00007f303e9e8301
__cxa_throw 0x00007f303e9e8535
fmt::v6::float_spec_handler::on_error format.h:2755
fmt::v6::internal::handle_float_type_spec<fmt::v6::float_spec_handler&> format.h:1190
fmt::v6::internal::basic_writer<fmt::v6::buffer_range<char> >::write_double<double, true> format.h:2765
fmt::v6::internal::arg_formatter_base<fmt::v6::buffer_range<char>, fmt::v6::internal::error_handler>::operator()<double, 0> format.h:1825
fmt::v6::visit_format_arg<fmt::v6::arg_formatter<fmt::v6::buffer_range<char> >, fmt::v6::basic_format_context<std::back_insert_iterator<fmt::v6::internal::buffer<char> >, char> > core.h:977
fmt::v6::internal::cf::vformat_to<fmt::v6::basic_format_context<std::back_insert_iterator<fmt::v6::internal::buffer<char> >, char>, fmt::v6::buffer_range<char>, const fmt::v6::internal::compiled_format<main()::<lambda()>::str, double> >(fmt::v6::buffer_range<char>, const fmt::v6::internal::compiled_format<main()::<lambda()>::str, double> &, fmt::v6::basic_format_args<fmt::v6::basic_format_context<std::back_insert_iterator<fmt::v6::internal::buffer<char> >, char> >) compile.h:225
fmt::v6::format<fmt::v6::internal::compiled_format<main()::<lambda()>::str, double>, double>(const fmt::v6::internal::compiled_format<main()::<lambda()>::str, double> &, const double &) compile.h:542
main main.cpp:62
__libc_start_main 0x00007f303e764b6b
_start 0x0000561b876865ba
  • Custom format specifiers crash with SIGSEGV:
constexpr auto f3 = fmt::compile<formattable>(FMT_STRING("{:!}\n"));
std::cout << fmt::format(f3, formattable{2});

This is from

fmt/include/fmt/compile.h

Lines 225 to 226 in 758446c

ctx.advance_to(
visit_format_arg(arg_formatter<Range>(ctx, nullptr, &specs), arg));
when the arg_formatter tries to dereference nullptr. Stack:

fmt::v6::basic_string_view<char>::begin core.h:309
fmt::v6::basic_parse_context<char, fmt::v6::internal::error_handler>::begin core.h:463
fmt::v6::formatter<formattable, char, void>::parse<fmt::v6::basic_parse_context<char, fmt::v6::internal::error_handler> > main.cpp:15
fmt::v6::internal::value<fmt::v6::basic_format_context<std::back_insert_iterator<fmt::v6::internal::buffer<char> >, char> >::format_custom_arg<formattable, fmt::v6::formatter<formattable, char, void> > core.h:772
fmt::v6::basic_format_arg<fmt::v6::basic_format_context<std::back_insert_iterator<fmt::v6::internal::buffer<char> >, char> >::handle::format core.h:917
fmt::v6::arg_formatter<fmt::v6::buffer_range<char> >::operator() format.h:2650
fmt::v6::visit_format_arg<fmt::v6::arg_formatter<fmt::v6::buffer_range<char> >, fmt::v6::basic_format_context<std::back_insert_iterator<fmt::v6::internal::buffer<char> >, char> > core.h:988
fmt::v6::internal::cf::vformat_to<fmt::v6::basic_format_context<std::back_insert_iterator<fmt::v6::internal::buffer<char> >, char>, fmt::v6::buffer_range<char>, const fmt::v6::internal::compiled_format<main()::<lambda()>::str, formattable> >(fmt::v6::buffer_range<char>, const fmt::v6::internal::compiled_format<main()::<lambda()>::str, formattable> &, fmt::v6::basic_format_args<fmt::v6::basic_format_context<std::back_insert_iterator<fmt::v6::internal::buffer<char> >, char> >) compile.h:225
fmt::v6::format<fmt::v6::internal::compiled_format<main()::<lambda()>::str, formattable>, formattable>(const fmt::v6::internal::compiled_format<main()::<lambda()>::str, formattable> &, const formattable &) compile.h:542
main main.cpp:56
__libc_start_main 0x00007fe92238db6b
_start 0x00005568559cc5ba
  • Longer custom format specifiers fail to compile but only if the compiled format is used in fmt::format or similar:
  constexpr auto f5 = fmt::compile<formattable>(FMT_STRING("{:!N}\n"));
  std::cout << fmt::format(f5, formattable{42});

Compiler output:

In file included from ../main.cpp:1:
/usr/local/include/fmt/compile.h: In instantiation of ?const fmt::v6::internal::format_part<typename fmt::v6::internal::char_t_impl<S>::type> (& fmt::v6::internal::compiled_format_base<S, typename std::enable_if<fmt::v6::is_compile_string<S>::value, void>::type>::parts() const)[fmt::v6::internal::compiled_format_base<S, typename std::enable_if<fmt::v6::is_compile_string<S>::value, void>::type>::num_format_parts] [with S = main()::<lambda()>::str; typename std::enable_if<fmt::v6::is_compile_string<S>::value, void>::type = void; fmt::v6::internal::compiled_format_base<S, typename std::enable_if<fmt::v6::is_compile_string<S>::value, void>::type>::parts_container = fmt::v6::internal::format_part<char> [2]; typename fmt::v6::internal::char_t_impl<S>::type = char]?:
/usr/local/include/fmt/compile.h:178:15:   required from ?typename Context::iterator fmt::v6::internal::cf::vformat_to(Range, CompiledFormat&, fmt::v6::basic_format_args<Context>) [with Context = fmt::v6::basic_format_context<std::back_insert_iterator<fmt::v6::internal::buffer<char> >, char>; Range = fmt::v6::buffer_range<char>; CompiledFormat = const fmt::v6::internal::compiled_format<main()::<lambda()>::str, formattable>; typename Context::iterator = std::back_insert_iterator<fmt::v6::internal::buffer<char> >]?
/usr/local/include/fmt/compile.h:542:36:   required from ?std::__cxx11::basic_string<Char> fmt::v6::format(const CompiledFormat&, const Args& ...) [with CompiledFormat = fmt::v6::internal::compiled_format<main()::<lambda()>::str, formattable>; Args = {formattable}; Char = char; typename std::enable_if<std::is_base_of<fmt::v6::internal::basic_compiled_format, CompiledFormat>::value, int>::type <anonymous> = 0]?
../main.cpp:71:47:   required from here
/usr/local/include/fmt/compile.h:304:54:   in ?constexpr? expansion of ?fmt::v6::internal::compile_to_parts<char, 2>(fmt::v6::to_string_view<main()::<lambda()>::str>(main()::<lambda()>::str{fmt::v6::compile_string()}))?
/usr/local/include/fmt/compile.h:272:30:   in ?constexpr? expansion of ?fmt::v6::internal::compile_format_string<true, char, fmt::v6::internal::compile_to_parts(fmt::v6::basic_string_view<Char>) [with Char = char; unsigned int N = 2]::<unnamed struct> >(format_str, collector)?
/usr/local/include/fmt/compile.h:157:36:   in ?constexpr? expansion of ?fmt::v6::internal::parse_format_string<true, char, fmt::v6::internal::format_string_compiler<char, fmt::v6::internal::compile_to_parts(fmt::v6::basic_string_view<Char>) [with Char = char; unsigned int N = 2]::<unnamed struct> > >(format_str, fmt::v6::internal::format_string_compiler<char, fmt::v6::internal::compile_to_parts(fmt::v6::basic_string_view<Char>) [with Char = char; unsigned int N = 2]::<unnamed struct> >(format_str, handler))?
/usr/local/include/fmt/format.h:2496:11:   in ?constexpr? expansion of ?(& handler)->fmt::v6::internal::format_string_compiler<char, fmt::v6::internal::compile_to_parts(fmt::v6::basic_string_view<Char>) [with Char = char; unsigned int N = 2]::<unnamed struct> >::on_format_specs((p + 1), end)?
/usr/local/include/fmt/compile.h:142:29: error: call to non-?constexpr? function ?void fmt::v6::internal::error_handler::on_error(const char*)?
     if (*it != '}') on_error("missing '}' in format string");
                     ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Example source file compiled with GCC 9.2 C++17:

#include <fmt/compile.h>
#include <fmt/format.h>
#include <iostream>

struct formattable {
  int i;
};

FMT_BEGIN_NAMESPACE
template <> struct formatter<formattable> : formatter<int> {
  bool special{false};

  template <typename ParseContext>
  constexpr auto parse(ParseContext &ctx) -> decltype(ctx.begin()) {
    if (*(ctx.begin()) == '!') {
      special = true;
      ctx.advance_to(ctx.begin() + 1);
      if (*(ctx.begin()) == 'N')
        ctx.advance_to(ctx.begin() + 1);
    }
    return formatter<int>::parse(ctx);
  }

  template <typename FormatContext>
  auto format(formattable f, FormatContext &ctx) -> decltype(ctx.out()) {
    if (special)
      ctx.out() = '~';
    return formatter<int>::format(f.i, ctx);
  }
};
FMT_END_NAMESPACE

int main() {
  // prints "00002" as expected
  std::cout << fmt::format(FMT_STRING("{:05}\n"), formattable{2});

  // prints "~2" as expected
  std::cout << fmt::format(FMT_STRING("{:!}\n"), formattable{2});

  // fails to compile because of get
  //  constexpr auto f0 =
  //      fmt::compile<formattable, formattable>(FMT_STRING("{}-{}\n"));
  //  std::cout << fmt::format(f0, formattable{2}, formattable{42});

  // prints "2", violates type safety
  constexpr auto f1 = fmt::compile<double>(FMT_STRING("{}\n"));
  std::cout << fmt::format(f1, formattable{2});

  // terminate called after throwing an instance of 'fmt::v6::format_error'
  //  what():  format specifier requires numeric argument
  //  constexpr auto f2 = fmt::compile<formattable>(FMT_STRING("{:05}\n"));
  //  std::cout << fmt::format(f2, formattable{2});

  // dereferencing a nullptr (fmt/compile.h:226)
  //  constexpr auto f3 = fmt::compile<formattable>(FMT_STRING("{:!}\n"));
  //  std::cout << fmt::format(f3, formattable{2});

  // fails to compile:
  //  std::cout << fmt::format(FMT_STRING("{:!}\n"), 5.3);
  // compiles:
  //  constexpr auto f4 = fmt::compile<double>(FMT_STRING("{:!}\n"));
  //  std::cout << fmt::format(f4, 42.0);

  // crashes at compile time only if both lines are uncommented
  //  constexpr auto f5 = fmt::compile<formattable>(FMT_STRING("{:!N}\n"));
  //  std::cout << fmt::format(f5, formattable{42});

  return 0;
}
@vitaut
Copy link
Contributor

vitaut commented Sep 23, 2019

Thanks for reporting. The constexpr format string compilation is more of a proof of concept at this moment. PRs would be very welcome.

@dkavolis
Copy link
Contributor Author

Removing constexpr and FMT_STRING has no effect on behaviour, I still get the same errors though at runtime.

@vitaut
Copy link
Contributor

vitaut commented Sep 25, 2019

Maybe the compiled format should contain template arguments that could later be used to enforce type safety at compile time but still allowing implicit conversions?

The compiled format already contains formatting argument types but doesn't enforce compatibility yet.

Format specifiers don't work with custom types

Yes, support for user-defined types is not implemented in compile.

@dkavolis
Copy link
Contributor Author

Thanks for the clarification. Any ideas if or when custom types will be supported in compile? Is it mostly replacing specs in

struct replacement {
arg_ref<Char> arg_id;
dynamic_format_specs<Char> specs;
};

with std::variant of all the formatters for the provided compile template type arguments? The few tagged unions in fmt could also benefit from std::variant by reducing code duplication (manually adding enum values and constructors).

The compiled format already contains formatting argument types but doesn't enforce compatibility yet.

My guess is that compile time enforcement wouldn't work with dynamic specs but at least it should be enforced at runtime.

@vitaut
Copy link
Contributor

vitaut commented Sep 29, 2019

Any ideas if or when custom types will be supported in compile?

It will be supported eventually but I don't have any specific dates.

Is it mostly replacing specs in ... with std::variant of all the formatters for the provided compile template type arguments?

Conceptually yes except that we cannot use variant for portability reasons.

My guess is that compile time enforcement wouldn't work with dynamic specs but at least it should be enforced at runtime.

Yes.

@vitaut
Copy link
Contributor

vitaut commented Jun 23, 2020

This should fixed now and all the examples here work as expected using the new FMT_COMPILE API:

#include <fmt/compile.h>
#include <fmt/format.h>
#include <iostream>

struct formattable {
  int i;
};

FMT_BEGIN_NAMESPACE
template <> struct formatter<formattable> : formatter<int> {
  bool special{false};

  template <typename ParseContext>
  constexpr auto parse(ParseContext &ctx) -> decltype(ctx.begin()) {
    if (*(ctx.begin()) == '!') {
      special = true;
      ctx.advance_to(ctx.begin() + 1);
      if (*(ctx.begin()) == 'N')
        ctx.advance_to(ctx.begin() + 1);
    }
    return formatter<int>::parse(ctx);
  }

  template <typename FormatContext>
  auto format(formattable f, FormatContext &ctx) -> decltype(ctx.out()) {
    if (special)
      ctx.out() = '~';
    return formatter<int>::format(f.i, ctx);
  }
};
FMT_END_NAMESPACE

int main() {
  // prints "00002" as expected
  std::cout << fmt::format(FMT_STRING("{:05}\n"), formattable{2});

  // prints "~2" as expected
  std::cout << fmt::format(FMT_STRING("{:!}\n"), formattable{2});

  // prints "2-42"
  std::cout << fmt::format(FMT_COMPILE("{}-{}\n"), formattable{2}, formattable{42});

  // prints "2"
  std::cout << fmt::format(FMT_COMPILE("{}\n"), formattable{2});

  // prints "~2"
  std::cout << fmt::format(FMT_COMPILE("{:!}\n"), formattable{2});

  // Fails to compile because of incorrect specifier.
  // std::cout << fmt::format(FMT_COMPILE("{:!}\n"), 5.3);

  // prints "~42"
  std::cout << fmt::format(FMT_COMPILE("{:!N}\n"), formattable{42});
}

Code generation for user-defined types is suboptimal but it will be addressed separately.

@vitaut vitaut closed this as completed Jun 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants