-
-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[question] Cant Parameterize and "Seperate out" Parser #46
Comments
Hi Did you consider putting whole (with ctpg::parser call) parser definitions into specialized classes? |
Hey thanks for the quick response
I suppose I don't quite understand why there is not static linkage? All my strings/arrays are declared static?
Is it possible to use CTRE in combination with your library?
A bit, but I am not sure how I would "merge/cat" the parsers together, if I have in the same bit of code an expression that has a float, and an expression that has a integer, but those are not the only things in the code (function declarations, etc.) don't I want to run all that through the same parser, rather than doing a pass for integers and a pass for floats with 2 different parsers? I suppose I can call additional parsers from within a parser, but that first parser would need awareness of all the possible types, I would also have to know when to stop, when to continue, and when to pass back too the "parent" parser. |
UPDATE @peter-winter : I managed to make a little progress using template specialization. It works if I call construct #include <iostream>
#include <functional>
#include <array>
#include <string>
#include <sstream>
#include <math.h>
#include <string.h>
#include <ctpg.hpp>
// This function was "yoiked" directly from https://github.com/peter-winter/ctpg //
constexpr size_t to_size_t( std::string_view integer_token )
{
size_t sum = 0;
for( auto digit : integer_token )
sum = ( sum * 10 ) + digit - '0';
return sum;
}
constexpr static const char parenthesis_scope_name_string[] = "ParenthesisScope";
constexpr static const char factor_name_string[] = "Factor";
constexpr static const char sum_name_string[] = "Sum";
enum class RegexLiteralTerms {
NaturalNumber = 0,
Identifier = 1
};
template< auto LiteralParameterType >
struct LiteralTerm
{
static_assert( true, "Error::Unkown literal type -- no valid specification" );
constexpr const static char regex[] = "";
constexpr const static char name[] = "Nothing";
constexpr const static auto term = ctpg::regex_term< regex >{ name };
constexpr const static auto literal_type = LiteralParameterType;
};
template<>
struct LiteralTerm< RegexLiteralTerms::NaturalNumber >
{
constexpr const static char regex[] = "[0-9][0-9]*";
constexpr const static char name[] = "NaturalNumber";
constexpr const static auto term = ctpg::regex_term< regex >{ name };
constexpr const static auto literal_type = RegexLiteralTerms::NaturalNumber;
};
template<>
struct LiteralTerm< RegexLiteralTerms::Identifier >
{
constexpr const static char regex[] = "[a-zA-Z\\_][a-zA-Z0-9\\_]*";
constexpr const static char name[] = "Identifier";
constexpr const static auto term = ctpg::regex_term< regex >{ name };
constexpr const static auto literal_type = RegexLiteralTerms::Identifier;
};
template< auto TypeParameterConstant
= RegexLiteralTerms::NaturalNumber >
struct DefaultTypes {
using Type = size_t;
};
template<>
struct DefaultTypes< RegexLiteralTerms::Identifier > {
using Type = std::string;
};
template<
auto TypeParameterConstant = RegexLiteralTerms::NaturalNumber,
typename CPlusPlusTypeParameterConstant
= DefaultTypes< TypeParameterConstant >::Type
>
struct Converter {
constexpr const static auto converter = to_size_t;
};
template<
typename OperhandTypeParameterType,
auto ConvertToTypeConstantParameter = Converter< RegexLiteralTerms::NaturalNumber >::converter,
typename LiteralRegexTermParameterConstant = LiteralTerm< RegexLiteralTerms::NaturalNumber >,
typename IdentifierRegexTermParameterConstant = LiteralTerm< RegexLiteralTerms::Identifier >,
auto FactorNameParameterConstant = parenthesis_scope_name_string,
auto SumNameParameterConstant = factor_name_string,
auto ParanthesisScopeParameterConstant = sum_name_string,
auto PlusCharacterParameterConstant = '+',
auto MinusCharacterParameterConstant = '-',
auto MultiplyCharacterParameterConstant = '*',
auto DivideCharacterParameterConstant = '/',
auto LeftParanethesisCharacterParameterConstant = '(',
auto RightParanethesisCharacterParameterConstant = ')'
>
struct ConstantExpression
{
constexpr static auto factor = ctpg::nterm< OperhandTypeParameterType >{ FactorNameParameterConstant };
constexpr static auto sum = ctpg::nterm< OperhandTypeParameterType >{ SumNameParameterConstant };
constexpr static auto parenthesis_scope = ctpg::nterm< OperhandTypeParameterType >{ ParanthesisScopeParameterConstant };
constexpr static auto nterms = ctpg::nterms(
factor, sum, parenthesis_scope
);
constexpr static auto plus_term = ctpg::char_term{
PlusCharacterParameterConstant,
1,
ctpg::associativity::ltor
};
constexpr static auto minus_term = ctpg::char_term{
MinusCharacterParameterConstant,
1,
ctpg::associativity::ltor
};
constexpr static auto multiply_term = ctpg::char_term{
MultiplyCharacterParameterConstant,
2,
ctpg::associativity::ltor
};
constexpr static auto divide_term = ctpg::char_term{
DivideCharacterParameterConstant,
2,
ctpg::associativity::ltor
};
constexpr static auto left_parenthesis_term = ctpg::char_term{
LeftParanethesisCharacterParameterConstant,
3,
ctpg::associativity::ltor
};
constexpr static auto right_parenthesis_term = ctpg::char_term{
RightParanethesisCharacterParameterConstant,
3,
ctpg::associativity::ltor
};
constexpr static auto terms = ctpg::terms(
plus_term,
minus_term,
multiply_term,
divide_term,
left_parenthesis_term,
right_parenthesis_term,
LiteralRegexTermParameterConstant::term,
IdentifierRegexTermParameterConstant::term
);
constexpr static auto rules = ctpg::rules(
factor( LiteralRegexTermParameterConstant::term ) >= ConvertToTypeConstantParameter,
factor( factor, multiply_term, IdentifierRegexTermParameterConstant::term )
>= []( size_t current_factor, auto, const auto& next_token ) {
return current_factor * ConvertToTypeConstantParameter( next_token );
},
factor( factor, divide_term, IdentifierRegexTermParameterConstant::term )
>= []( size_t current_factor, auto, const auto& next_token ) {
return current_factor / ConvertToTypeConstantParameter( next_token );
},
parenthesis_scope( left_parenthesis_term, factor, right_parenthesis_term )
>= [] ( auto, auto factor, auto ) { return factor; },
factor( parenthesis_scope ) >= []( auto parenthesis_scope ) { return parenthesis_scope; },
factor( factor, multiply_term, parenthesis_scope )
>= []( auto factor, auto, auto parenthesis_scope ) {
return factor * parenthesis_scope;
},
factor( factor, divide_term, parenthesis_scope )
>= []( auto factor, auto, auto parenthesis_scope ) {
return factor / parenthesis_scope;
},
factor( sum ) >= []( auto sum ) { return sum; },
sum( factor, plus_term, factor )
>= []( auto current_sum, auto, const auto& next_token ) {
return current_sum + next_token;
},
sum( factor, minus_term, factor )
>= []( auto current_sum, auto, const auto& next_token ) {
return current_sum - next_token;
}
);
constexpr const static auto parser = ctpg::parser<
decltype( factor ),
decltype( terms ),
decltype( nterms ),
decltype( rules ),
ctpg::use_generated_lexer
>{
factor,
terms,
nterms,
rules,
};
}; I can now parameterize... I think, however, I still cant enter the necessary components into the parser at the end there, it keeps telling me there are too many initializers. Here is a gist of the output (I ran it through clang-format 😃 ) My goal is to |
Well, I tink there are, I don't think you can initialize parser using {} initializer with 4 arguments. It is not a trivially initalized class, you have to use a constructor with the () syntax. You don't specify the template parameters, the deduction guides take care of this. Wouldn't this work:
|
Just tried it with/without e.g Compiler keeps complaining that there are too many initializes. Also Compiler/Platform: |
I was able to resolve this by creating a member function that returns struct MyParser
{
//... define terms etc.
template<auto OperationParameterConstant>
constexpr static const auto binary_operation_rules(auto to, auto left_term, auto operator, auto right_term) {
return ctpg::rules(
to(left_term, operator, right_term) >= OperationParameterConstant
);
}
consteval static const auto rules()
{
return ctpg::rules(
math_term(integer) >= [](auto integer) { return term{integer}; },
std::tuple_cat(
binary_operation_rules<[](auto left, auto add, auto right) { return left + right; }>(
sum, sum, plus, math_term
),
binary_operation_rules<[](auto left, auto add, auto right) { return left + right; }>(
sum, sum, plus, math_term
),
)
);
}
};
constexpr static const auto my_parser = ctpg::parser(
MyParser::expression,
MyParser::my_terms,
MyParser::my_non_terminal_terms,
MyParser::rules()
); |
Cool! |
I am trying to split apart (and parameterize) my parser, something like this..
This way, for multiple data-types I can make more constant expresssion regexes for different data-types (and for different types of literals with different regexes e.g natural number
[0-9][0-9]*
vs float[0-9]*.[0-9]+
).The problem comes when I try to use a literal, it seems to have trouble accepting the string for the regex in the
literal_term
I have tried multiple methods, at first all the parameters were just plain auto, and I tried serializing the strings into templates, then injecting them back into an array type with the size known at compile time (as a bunch of errors with the size of the type not being known at compile time come up otherwise).
I would then pass
natural_number_regex
into ConstantExpressionIf I do it like this
The compiler would spit out a whole bunch of template barf I have spent a while formatting and trying to make some sense of, clang is giving me something GCC wont
These are just a few errors/warnings/notes I thought may be relevant
I have also tried specifying
auto&
(because I saw it inctpg::buffers::cstring_buffer
) instead of justauto
-- no diceWith either, if I specify raw literals in the parameters I get something like
If I try to substitute constants such as
I get right back to the errors I had before
I have no idea how to make this work properly and think I may be forced to go back to a "monolithic" parser. Anyone know how to make this work
Thank you,
P.s Here is a gist of a full compiler output and here is my header file and this is a Main.cpp using it, if you comment out the second rule, it works because it does not have the literal term
P.s.s If I substitute the mock rules with the actual ones I want too use, I get
This warning is showing up quite persistently.
The text was updated successfully, but these errors were encountered: