PipelineC++

[suarezvictor]

Idea: support some C++ features.
Example instead of uint12_5_2(x) (extract bits 5 to 2 of a 12 bit value) one can write: bits<5,2>(x)

No need to address all C++ syntax but this way it would be better to keep a source compiled with a regular compiler and would be easier to parametrize constants. For example:
#define BITS 10
bits<BITS,0>(x);

[JulianKemmerer]

I am all for supporting more advanced (i.e. C++) syntax.

However, some initial reasons why it doesn't exist already:

1. I dont know how to modify the pycparser I am using to support C++ style '<' '>' tokens for template types, etc.
2. Less importantly, it was originally done as text 5_2 in the function name also because I didnt want people to think it could be used with run-time variable values. I.e. compile time 'select bits 5:2' is different from run time 'select bits M:N' (where M expression evals to 5, etc). So didnt want to do like bits(5,2)(x) with macros or something. But looking back, its possible to implement the 'select bits M:N' run time variable - its pretty much a random access looking thing like I have dealt with for arrays already.

Like I said, I would be happy to support this I just literally dont know how at the moment syntax wise + a little bit of complication+implementation needed for random access case (I suppose template <> values are compile time constant anyway, dont need to worry about random case yet).

[makslevental]

project - my google search for parsing cpp with python all come up with clang,pycparser...

yea it wasn't the first thing because googling c++ parser gets you parsers written in c++. i googled "grammer python c++" and the like. thinking a little bit more on it, possibly the right way is to use antlr, which is mature and has a cpp grammar and will generate a python runtime (which i guess are just python bindings to the java runtime). i say possibly because while my intuition tells me antlr is more mature than tree-sitter, i don't know convenient it is today (it's a pretty old project).

What is the mechanism for resolving if, for ex. someone writes bits<Param1+Param2,0>(x) where the Params are non-type template parameter values. Where eventually the + resolves to known constant integers bits<10,0>(x) maybe

simple cases of this are variously called constant propagation and common sub-expression elimination. i don't know how other compilers do it but llvm does it at the ir level 1, 2. mlir does it as well 1 2 (and thus does CIRCT). more complicated cases are variously called symbolic execution, abstract interpretation. i spend most of my time in DNN compiler land and in that area i've used symbolic execution to get shapes of tensors (ie i've pitched in here).

There is ~bare minimum compile time constant folding/calculating/propagating in PipelineC - it doesnt do cross hierarchy stuff or struct/array data types... and its very rough to maintain/slowly implemented... I couldnt imagine extending that to really support the full extent of compile time calculations needed to resolve C++ stuff?

indeed in my own "generator framework" i have the same thing but i wouldn't want to support all the kinds of things templates support (because then you'd basically just be reimplementing C++ with different semantics).

[suarezvictor]

Most things solvable at compile time are certainly solved by a C++ compiler. Clang for example is pretty smart, it can also discover the results of some simple algorithms involving loops and the like. Everything consteval will certainly be solved at compile time, while not warranted by constexpr.
For parsing C++ we have my python parser based on clang's cindex ;-) see CFlexHDL project

[JulianKemmerer]

I think even if we did pick up the fancier/heavier LLVM/clang parsing infrastructure...correct me if Im wrong... I dont think at the parsing layer it would be handling constant propagation and common sub-expression and the like, especially regards to compile time constant template parameter computations. @suarezvictor does CFlexHDL do anything is that regard?

I'm afraid just supporting the syntax isnt enough to get that functionality - but needing further analysis/passes/conversion to CPU specific IRs for working with...

Like I am seeing thsi as either:
A) easy hacky parsing of template looking syntax without any fancy compile time computations - just enough to ~look like cpp but really limited in functionality
B) Use a full parser+IR stack dedicated to hardware, like XLS, or CIRCT which have done the hard work already

[JulianKemmerer]

Saw this today
https://github.com/aappleby/Metron
Metron C++ to Verilog Translator

[JulianKemmerer]

How Metron emphasizes a very limited subset of C++ into a very limited subset of Verilog makes me feel like it would be valuable to start with above A) easy hacky parsing of template the hacky barely working ~ parses like C++ templates but isnt fully handled C++ templates

That said - how hard would it be to add just like dumbest layer of bits<Y,X>(num) syntax parsing to pycparser? Like @makslevental pointed out to start - just give me whatever strings (AST nodes?) are "Y" and "X" and hack from there

[suarezvictor]

I proppsed such template syntax of kind bits<5,2> doesn't support runtime variables but compile time calculated constant so I seem then very suitable for such cases. Using non constant expressions will raise an error in a C++ compiler and pipelineC can mimic such behavior El dom., 3 oct. 2021 14:59, Julian Kemmerer ***@***.***> escribió:

…

I am all for supporting more advanced (i.e. C++) syntax. However, some initial reasons why it doesn't exist already: 1. I dont know how to modify the pycparser <https://github.com/eliben/pycparser> I am using to support C++ style '<' '>' tokens for template types, etc. 2. Less importantly, it was originally done as text 5_2 in the function name also because I didnt want people to think it could be used with run-time variable values. I.e. compile time 'select bits 5:2' is different from run time 'select bits M:N' (where M express evals to 5, etc). So didnt want to do like bits(5,2)(x) with macros or something. But looking back, its possible to implement the 'select bits M:N' run time variable - its pretty much a random access looking thing like I have dealt with for arrays already. Like I said, I would be happy to support this I just literally dont know how at the moment syntax wise + a little bit of complication+implementation needed for random access case (I suppose template <> values are compile time constant anyway, dont need to worry about random case yet). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#32 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBHVWJIL26AIALDRHOY6G3UFCKZJANCNFSM5FHSMUXA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[JulianKemmerer]

@suarezvictor Do you know about parsers 'lexers' yacc syntax trees etc? things https://github.com/eliben/pycparser deals with?

That github recommends doing something different for parsing C++ (recommends LLVM based I think). But maybe we could modify the more simply pycparser to just add support for compile time constant bits<5,2> kind of syntax without doing the rest of C++ (yet?)

[suarezvictor]

Definitely, I don't suggest supporting all C++ syntax. Just some tiny portion that adds such functions. For the moment I suggest just the <> operator for certain functions. Having calculated compile-time constantes instead of numbers in symbol names will be a great addition. There are manu good parsers to support needed features. Let me know if you want me to collaborate selecting one. El dom., 3 oct. 2021 23:34, Julian Kemmerer ***@***.***> escribió:

…

@suarezvictor <https://github.com/suarezvictor> Do you know about parsers 'lexers' yacc syntax trees etc? things https://github.com/eliben/pycparser deals with? That github recommends doing something different for parsing C++ (recommends LLVM based I think). But maybe we could modify the more simply pycparser to just add support for compile time constant bits<5,2> kind of syntax without doing the rest of C++ (yet?) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#32 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBHVWPX45U77WFK3C2HSMTUFEHDNANCNFSM5FHSMUXA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[JulianKemmerer]

I suppose its not just collaborate selecting one but I need some serious help even understanding "how to go from C-code to python C-abstract syntax tree objects" (i.e how does pycparser work)
From pycparser outputs, the C-AST to traverse, I know how to proceed.
I dont know how to modify pycparser - and 'selecting a new parser' sounds like lots and lots of work. I am sort of tied to pycparser until I understand how to modify it or can fully replace it 🤷

[bartokon]

What about LLVM? (I did not read whole topic, so idk)

[suarezvictor]

To me the issue with LLVM is that it's designed to CPU so most operations are limited to data widths are a power of 2 of 8-bit bytes so not really suitable for arbitrary bit widths as normally used for hardware design. Otherwise it's a really good tool

[suarezvictor]

I can definitely help with that. Maybe I have more experience in software as I see you have more experience in hardware design than me. For exampĺe a project I did (with success) was to parse matlab code and translate it to python with bit exact results for doing math processing (for algorithms of hundreds of lines) Again I'm time limited but from time to time, I'll have some.

…

On Mon, Oct 4, 2021 at 1:56 PM Julian Kemmerer ***@***.***> wrote: I suppose its not just collaborate selecting one but I need some serious help even understanding "how to go from C-code to python C-abstract syntax tree objects" (i.e how does pycparser work) From pycparser outputs, the C-AST to traverse, I know how to proceed. I dont know how to modify pycparser - and 'selecting a new parser' sounds like lots and lots of work. I am sort of tied to pycparser until I understand how to modify it or can fully replace it 🤷 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#32 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBHVWMA2GO722F3TAL7C2LUFHMENANCNFSM5FHSMUXA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[suarezvictor]

One possible solution is to parse the file as C++ (considering that C is, in practice, a subset of C). For that you can call clang to analyze the file. You can call clang libs (notthe executable) from python and have the AST in python. An example of that is here: https://sudonull.com/post/907-An-example-of-parsing-C-code-using-libclang-in-Python this was a quick search and the topic can't be solved in just minutes, but the point is that there exist lots of similar solutions that we can reuse. A main thing to solve is first to have a _design_ of what features would be useful to incorporate. The first one was the bit access as I proposed, we may start discovering more as we use them. A feature I see really useful (and that doesn't need C++ syntax) is to have the new types "uX_t" meaning "integer of at least X bits", then the type "bool" would be equal to "u1_t", and implemented as 1 bit in hardware and bit-exact simulation but as as real bool in raw C compiling. Let's start to design some more desired features, that will influence the changes to the parser needed. In regards to the bit extract function, for example, uint12_3_2, what does the first number mean? I think if we're asking for two bits, the results should be a uint2_t. And in case the operand is of no more than 2 bits, be rejected or zero-padded and a warning issued. On Mon, Oct 4, 2021 at 2:04 PM Victor Suarez Rovere ***@***.***> wrote:

…

I can definitely help with that. Maybe I have more experience in software as I see you have more experience in hardware design than me. For exampĺe a project I did (with success) was to parse matlab code and translate it to python with bit exact results for doing math processing (for algorithms of hundreds of lines) Again I'm time limited but from time to time, I'll have some. On Mon, Oct 4, 2021 at 1:56 PM Julian Kemmerer ***@***.***> wrote: > I suppose its not just collaborate selecting one but I need some serious > help even understanding "how to go from C-code to python C-abstract syntax > tree objects" (i.e how does pycparser work) > From pycparser outputs, the C-AST to traverse, I know how to proceed. > I dont know how to modify pycparser - and 'selecting a new parser' sounds > like lots and lots of work. I am sort of tied to pycparser until I > understand how to modify it or can fully replace it 🤷 > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#32 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACBHVWMA2GO722F3TAL7C2LUFHMENANCNFSM5FHSMUXA> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > >

[JulianKemmerer]

Regarding "integer of at least X bits". I think that sounds good to me.
But how does that effect rollover behavior? Hardware people expect, ex. +1 in a loop on a uint2_t , 0b00, 0b01, 0b10, 0b11, 0b00, repeat. I.e. no need for extra bounds checking. I think signed numbers get tricky too.

[JulianKemmerer]

uint12_3_2 you can get some information on in the wiki https://github.com/JulianKemmerer/PipelineC/wiki/Automatically-Generated-Functionality#bitmanip.

uint12 specified the input to the function. Bit widths larger than uint12 are truncated to uint12. Bit widths smaller than uint12 are zero extended (or sign extended, can get a little tricky). Can add warnings about this sure.

From those 12 bits, in HDL traditionally written as 11:0 or 11 downto 0 select bits 3 downto 2 or 3:2. This yields a 2 bit number uint2_t as you expected.

[suarezvictor]

Regarding the "integers of at least N bits" muy proposal is that should be common to mix both types on a same program. When you exactly two bits, you use uint2_t. And when you can live with "at least 2 bits" you use u2_t. This will only affect simumation performance, since synthesis will use the exact bit number. And in simulation you can select both behaviors (one will be faster and the other will be more exact)
That will help compatibility with C, as an added feature. I strongly think that most algorithms could be specified with the "al least N" bits types.

[suarezvictor]

For users that for any reason don't like the feature, they can always use the bit exact version for all types. Folks like me will prefer gain un simulatuon speed by using the flexible types. And al this isn't needed as a change of the core language, it's only a bunch of typedefs that can be optionally included as a header when desired. But I think promoting the possibilities will be valuable.

[JulianKemmerer]

Oh I see - there being a difference between uint2_t and u2_t.

Does C already have standard concepts for 'exactly N bits' vs. N+ bits? I forget how for example. uint32_t is defined - I think its defined as 'at least 32bs?' Maybe I am using uintN_t types wrong.

[suarezvictor]

In regards to the "uint12_3_2" that will first extend/truncate to 12 bits, it seems in majority of cases the first number is not needed. For example, if operand is unsigned, taking bits 3 to 2 will produce same results for operands of any size regardless the "12". That is, function " uintX_3_2" will be all equal for any X. In case of signed operands, the behavior will be such confusing that maybe it will be convenient to first do the size conversion and then the bit extracción with two function calls (and I think such signed operations will be uncommon)
My proposal is that, if some users will like the current design, it can be left there but other set of functions not needing the first number should be provided. I personally will prefer the versions with just two numbers and call two functions for the unusual case

[suarezvictor]

(Please consider my proposals are in view of ease of compatibilty with C and helper functions te be developed to achieve simulation by just compiling the source)

[JulianKemmerer]

Yeah I understand and that makes alot of sense especially for the unsigned case.

So its currently:
uint<Y-X+1>_t <type_prefix>_Y_X(<type> val)
ex.uint2_t uint12_3_2(uint12_t val)

Maybe something like this for 'assumed' input size based on the bit range. Make it so the uint12 isnt needed

uint<Y-X+1>_t bits_Y_X(uint<Y+1>_t val)
ex.uint2_t bits_3_2(uint4_t val)
^Meaning the uint12_t is first truncated to 4 bits, and then 3 downto 2 is selected
Which could be macro'd for now as
uint2_t b = BITS(val, 3, 2);

[suarezvictor]

I agree on everything, and liked your macro BITS(val, 3, 2). Maybe BITS(3, 2, val)? See below. Additionally, I can provide a more syntax-sugarish implementation for C++ compilers like clang with a syntax as previously proposed: bits<3,2>(val), equivalent to your macro. And hopefully we can tweak or replace the parser to support that. The benefit is that with both forms we can use compile-time constants that are next to a "must" to provide some flexibility. Using template trickery (that I can write) we can offer some compile-time calculations like detecting the most significant bit of a constant, bit count, clog2 equivalent, "all ones" of some size, etc. In regards to types, in C you have the basic types "char, int, long". Their size is platform-dependent and never defined. In C++ you have the additional "bool" value which is also platform dependent (you don't know how many bits it has nor the values used, but in practice is like a char). Then there are nonstandar compiler extensions like "long long" and the like. Since such definitions has platform dependence (actually "indefinitions"), there are handy typedefs in headers like stdint.h which provides uint32_t and the like fixed-size types. They're fixed size, no possible extensions. The extensibility proposed is mine (surely there are others that suggested the same but I'm unaware).

…

On Mon, Oct 4, 2021 at 8:40 PM Julian Kemmerer ***@***.***> wrote: Yeah I understand and that makes alot of sense especially for the unsigned case. So its currently: uint<Y-X+1>_t <type_prefix>_Y_X(<type> val) ex.uint2_t uint12_3_2(uint12_t val) Maybe something like this for 'assumed' input size based on the bit range. Make it so the uint12 isnt needed uint<Y-X+1>_t bits_Y_X(uint<Y+1>_t val) ex.uint2_t bits_3_2(uint4_t val) Which could be macro'd for now as uint2_t b = BITS(val, 3, 2); — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#32 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBHVWLHNLSAXPGNMXRNPPTUFI3ONANCNFSM5FHSMUXA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[suarezvictor]

auto keyword proposal. The "auto" keyword from C++ can be copied, for example: uint2_t a, uint2_t b; auto c = a+b; Then c will be uint3_t This would be useful for lots of things, for example for floating point operator implementations that depends on compile-time calculated constants, so the same algorithm is written only once for single and double precision or also arbitrary exponent and mantissa precision. This idea like others are again here for the a future implementation. El lun., 4 oct. 2021 20:53, Victor Suarez Rovere ***@***.***> escribió:

…

I agree on everything, and liked your macro BITS(val, 3, 2). Maybe BITS(3, 2, val)? See below. Additionally, I can provide a more syntax-sugarish implementation for C++ compilers like clang with a syntax as previously proposed: bits<3,2>(val), equivalent to your macro. And hopefully we can tweak or replace the parser to support that. The benefit is that with both forms we can use compile-time constants that are next to a "must" to provide some flexibility. Using template trickery (that I can write) we can offer some compile-time calculations like detecting the most significant bit of a constant, bit count, clog2 equivalent, "all ones" of some size, etc. In regards to types, in C you have the basic types "char, int, long". Their size is platform-dependent and never defined. In C++ you have the additional "bool" value which is also platform dependent (you don't know how many bits it has nor the values used, but in practice is like a char). Then there are nonstandar compiler extensions like "long long" and the like. Since such definitions has platform dependence (actually "indefinitions"), there are handy typedefs in headers like stdint.h which provides uint32_t and the like fixed-size types. They're fixed size, no possible extensions. The extensibility proposed is mine (surely there are others that suggested the same but I'm unaware). On Mon, Oct 4, 2021 at 8:40 PM Julian Kemmerer ***@***.***> wrote: > Yeah I understand and that makes alot of sense especially for the > unsigned case. > > So its currently: > uint<Y-X+1>_t <type_prefix>_Y_X(<type> val) > ex.uint2_t uint12_3_2(uint12_t val) > > Maybe something like this for 'assumed' input size based on the bit > range. Make it so the uint12 isnt needed > > uint<Y-X+1>_t bits_Y_X(uint<Y+1>_t val) > ex.uint2_t bits_3_2(uint4_t val) > Which could be macro'd for now as > uint2_t b = BITS(val, 3, 2); > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#32 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACBHVWLHNLSAXPGNMXRNPPTUFI3ONANCNFSM5FHSMUXA> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > >

[suarezvictor]

I think I'm closer to template parsung or any C++ construct using Clang's cindex than using a custom parser with unknown bugs... El vie., 27 may. 2022 22:40, Julian Kemmerer ***@***.***> escribió:

…

How Metron emphasizes a very limited subset of C++ into a very limited subset of Verilog makes me feel like it would be valuable to start with above A) easy hacky parsing of template the hacky barely working ~ parses like C++ templates but isnt fully handled C++ templates That said - how hard would it be to add just like dumbest layer of bits<Y,X>(num) syntax parsing to pycparser? Like @makslevental <https://github.com/makslevental> pointed out to start - just give me whatever strings are "Y" and "X" and hack from there — Reply to this email directly, view it on GitHub <#32 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBHVWMAD5SPO2HF4G6OBLLVMF2PHANCNFSM5FHSMUXA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[JulianKemmerer]

Yeah the python bindings you have demoed for using clang parsing does seem to be working great - probably the best option 😎

[JulianKemmerer]

@suarezvictor at one point you sent a snippet like

import sys
import clang.cindex
from clang.cindex import TokenKind

index = clang.cindex.Index.create()
tu = index.parse(sys.argv[1])
for t in tu.get_tokens(extent=tu.cursor.extent):
    if t.kind != TokenKind.COMMENT:
      print(t.spelling, "\t//", t.kind)

That gives tokens correctly - but do you know how to just print the entire AST? You just visit each child right?

I tried doing a recursive, etc

def visit(node):
    print(node.spelling)
    for c in node.get_childen():
       visit(c)

but doesnt seem to fully visit all the children of the AST? (seems to be stuck at a fund def open close {} compound statement?) Am I doing something dumb?