WIP: CHG: Use pycparser as a git submodule #159

AlexLao512 · 2023-02-04T22:51:02Z

WIP: Needs more testing.

Use pycparser as submodule.

JulianKemmerer · 2023-02-05T02:01:14Z

IIUC this does not pin a version of pycparser - and would, as coded now, have people doing git init recursive clone stuff and would pull the latest pycparser code
@Datavenia does this work with the direction/mechanism you were thinking for pinning versions?

ghost · 2023-02-05T02:40:03Z

versions?

We could add a pinned version to the pyproject.toml, this means people would need to use poetry as a user but it's probably the most extensible system.

AlexLao512 · 2023-02-05T03:44:24Z

IIUC this does not pin a version of pycparser - and would, as coded now, have people doing git init recursive clone stuff and would pull the latest pycparser code @Datavenia does this work with the direction/mechanism you were thinking for pinning versions?

It actually does pin, there is a .gitmodules file with the URL to the repository with a release tag (version).

AlexLao512 · 2023-02-05T03:58:22Z

Sorry, some mistakes above, the commit on the remote repo is pinned here. If you click through to the commit you will see the release tag.

If is in the submodule itself via commit hash and not in the .gitmodules branch = because the remote repo has a TAG on master, instead of a release branch and tag.

AlexLao512 · 2023-02-05T04:01:02Z

versions?

We could add a pinned version to the pyproject.toml, this means people would need to use poetry as a user but it's probably the most extensible system.

I suggest we use a setup.py so pip can be used. pip is installed along with the users Python install, no additional pieces needed on the users end.

JulianKemmerer · 2023-02-13T03:56:41Z

It sounds like doing like this PR doing, and using git submodules to pull in pycparser at a certain commit point ~essentially having a copy of the repo inside pipelinec repo like now but the support git way - makes sense to do. I of course want to test it but sounds simple and easy.

And then after, in a separate PR/issue perhaps, is the question of 'proper' dependency/package management for python projects? And IIUC it has to be either or pip or poetry based management? Idk if it can be both easily?

But I am thinking this PR can go through sooner than deciding on / setting up pip/poetry with pycparser dependency etc

AlexLao512 · 2023-02-13T03:59:53Z

Yea so this does half the cleanup required for moving to proper Python package management, the submodule part is trivial to remove later. If you want to test, I think this can be merged.

JulianKemmerer · 2023-02-13T04:10:00Z

Yeah testing I want to do is basic - does it still seem to parse c code and not crash kinda tests

Should have some time this week

JulianKemmerer · 2023-02-14T01:36:39Z

Wow I just realized you got the pycparser 2.18 tagged commit version that should exactly match what I had copied into the repo - tests should go smoothly 👍

JulianKemmerer · 2023-02-14T03:11:30Z

IIUC

New repo clone will need to be git clone --recurse-submodules

And users pulling to update their repo need to one time update+init submodules

git pull
git submodule update --init --recursive

JulianKemmerer · 2023-02-14T03:32:13Z

@AlexLao512 can you change the temp work around locations in code to use the absolute path to the repo?

from utilities import REPO_ABS_DIR

# TODO: Temporarily import from submodule, remove this hack when we create a proper pipelinec setup.py
sys.path.append(REPO_ABS_DIR()+'/submodule/pycparser')

Otherwise depends on being in ./src/ as is on branch

If I wanted to make/suggest this change I would need to clone your fork of my repo w/ the branch - and then make another branch to act as a PR into your branch? 😬

JulianKemmerer · 2023-02-22T22:31:06Z

Oh dope - I didnt know I could just commit and push to a repo owned by not my user (I guess forking my repo gives me access by default idk?)

So I think I can continue to make the changes need to the PR for full merge 👍 hopefully sooner than later...

JulianKemmerer · 2023-02-23T01:52:28Z

Something fucky is up

Trying the branch I get a C parsing related ~'nodes in a dictionary arent as expected' internal pipelinec sanity check error (tried examples/blink.c)

Weird...

C_AST_VAL_UNIQUE_KEY_DICT_MERGE Dicts aren't unique: {'BIN_OP_EQ[blink_c_l17_c6_e220]': <pycparser.c_ast.BinaryOp object at 0x7f341b2f3310>, 'TRUE_CLOCK_ENABLE_MUX[blink_c_l18_c1_af86]': <pycparser.c_ast.Compound object at 0x7f341b2e8180>, 'FALSE_CLOCK_ENABLE_MUX[blink_c_l25_c1_5b0e]': <pycparser.c_ast.Compound object at 0x7f341b2e8040>, 'UNARY_OP_NOT[blink_c_l20_c12_cefe]': <pycparser.c_ast.UnaryOp object at 0x7f341b4baa00>} {'BIN_OP_EQ[blink_c_l17_c6_e220]': <pycparser.c_ast.BinaryOp object at 0x7f341b2f3220>, 'TRUE_CLOCK_ENABLE_MUX[blink_c_l18_c1_af86]': <pycparser.c_ast.Compound object at 0x7f341b1cc7c0>, 'FALSE_CLOCK_ENABLE_MUX[blink_c_l25_c1_5b0e]': <pycparser.c_ast.Compound object at 0x7f341b1ccc40>, 'BIN_OP_PLUS[blink_c_l26_c5_c986]': <pycparser.c_ast.BinaryOp object at 0x7f341b2f3630>}
key BIN_OP_EQ[blink_c_l17_c6_e220]
v1 <pycparser.c_ast.BinaryOp object at 0x7f341b2f3310> blink_c_l17_c6_5b93
v2 <pycparser.c_ast.BinaryOp object at 0x7f341b2f3220> blink_c_l17_c6_72f9
Traceback (most recent call last):
  File "./src/pipelinec", line 109, in <module>
    parser_state = C_TO_LOGIC.PARSE_FILE(c_file)
  File "/home/julian/Desktop/AlexLao512/PipelineC/src/C_TO_LOGIC.py", line 9723, in PARSE_FILE
    parser_state.FuncLogicLookupTable = APPEND_FUNC_NAME_LOGIC_LOOKUP_TABLE(
  File "/home/julian/Desktop/AlexLao512/PipelineC/src/C_TO_LOGIC.py", line 10967, in APPEND_FUNC_NAME_LOGIC_LOOKUP_TABLE
    logic = C_AST_FUNC_DEF_TO_LOGIC(
  File "/home/julian/Desktop/AlexLao512/PipelineC/src/C_TO_LOGIC.py", line 8807, in C_AST_FUNC_DEF_TO_LOGIC
    body_logic = C_AST_NODE_TO_LOGIC(
  File "/home/julian/Desktop/AlexLao512/PipelineC/src/C_TO_LOGIC.py", line 1901, in C_AST_NODE_TO_LOGIC
    return C_AST_COMPOUND_TO_LOGIC(c_ast_node, prepend_text, parser_state)
  File "/home/julian/Desktop/AlexLao512/PipelineC/src/C_TO_LOGIC.py", line 5188, in C_AST_COMPOUND_TO_LOGIC
    item_logic = C_AST_NODE_TO_LOGIC(
  File "/home/julian/Desktop/AlexLao512/PipelineC/src/C_TO_LOGIC.py", line 1905, in C_AST_NODE_TO_LOGIC
    return C_AST_IF_TO_LOGIC(c_ast_node, prepend_text, parser_state)
  File "/home/julian/Desktop/AlexLao512/PipelineC/src/C_TO_LOGIC.py", line 5997, in C_AST_IF_TO_LOGIC
    true_logic.MERGE_COMB_LOGIC(false_logic)
  File "/home/julian/Desktop/AlexLao512/PipelineC/src/C_TO_LOGIC.py", line 762, in MERGE_COMB_LOGIC
    self.submodule_instance_to_c_ast_node = C_AST_VAL_UNIQUE_KEY_DICT_MERGE(
  File "/home/julian/Desktop/AlexLao512/PipelineC/src/C_TO_LOGIC.py", line 315, in C_AST_VAL_UNIQUE_KEY_DICT_MERGE

AlexLao512 · 2023-02-25T05:27:31Z

Might be a good idea to diff the version you have in the repo and the tagged releases of pycparser.

JulianKemmerer · 2023-04-26T23:04:48Z

Hopefully I can get around to this in the next week or so 👍 , starting with the diff

JulianKemmerer · 2023-04-27T23:01:51Z

OK I think I know whats going on.

The version of the files I copied into my repo in ~2017/2018 some time was marked at version 2.18. And that is what was used for the submodule.

However, the files are marked as version 2.18 all the way through to 2.19. Meaning, I likely had copied the files after 2.18 release, including any differences accumulated since but before 2.19.

Anywhoo
if you look here
https://github.com/eliben/pycparser/commits/caa4c11ebb99ed5cf854dc6342b5352d5ff52686

The commit right before 2.19 is eliben/pycparser@bcb5f9f

I checked out that commit on top of this PR and it seems to be working fine in early tests...

JulianKemmerer · 2023-04-27T23:05:14Z

Perhaps it would be sane to try a version of this PR using exactly version 2.19 which differs by only the version bump commit 😅

JulianKemmerer · 2023-04-28T20:48:35Z

Now thats the submodule is up to 2.19 again the last thing seems to be fixing the path append like

from utilities import REPO_ABS_DIR

# TODO: Temporarily import from submodule, remove this hack when we create a proper pipelinec setup.py
sys.path.append(REPO_ABS_DIR() + '/submodule/pycparser')

JulianKemmerer · 2023-04-30T17:43:59Z

Looking great so far, a test build running...

But I pushed another minor fix to the pipelinec main repo

How do I get that change onto this branch under test - you mentioned 'rebase' is what I want? - ill do some googling 😏

AlexLao512 · 2023-04-30T23:34:05Z

Done. You don't actually have push access to my fork. I can pull changes from your forks master branch into my forks master branch, then I can rebase my forks branch onto my forks master branch (which is now in sync with yours).

JulianKemmerer · 2023-05-01T23:44:55Z

I have more testing to do than I anticipated

Using the new pycparser version I am getting designs with ~10% more LUTs

Trying to decide how to debug this or just accept the hit for now...

AlexLao512 force-pushed the submodule_pycparser branch from 6fbd60f to 8d0cc17 Compare February 5, 2023 03:56

JulianKemmerer mentioned this pull request Feb 16, 2023

added pre-commit hook #161

Closed

JulianKemmerer self-assigned this Apr 26, 2023

AlexLao512 force-pushed the submodule_pycparser branch from 9cb6c7d to 51119ac Compare April 28, 2023 02:10

AlexLao512 added 3 commits April 30, 2023 19:32

CHG: Use pycparser as a git submodule

3dd31bd

CHG: Upgrade pycparser to 2.19

08e82e2

FIX: Use root directory of repository instead of relative path

ccf1322

AlexLao512 force-pushed the submodule_pycparser branch from 2391519 to ccf1322 Compare April 30, 2023 23:32

JulianKemmerer removed their assignment May 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: CHG: Use pycparser as a git submodule #159

WIP: CHG: Use pycparser as a git submodule #159

AlexLao512 commented Feb 4, 2023

JulianKemmerer commented Feb 5, 2023

ghost commented Feb 5, 2023

AlexLao512 commented Feb 5, 2023 •

edited

AlexLao512 commented Feb 5, 2023 •

edited

AlexLao512 commented Feb 5, 2023

JulianKemmerer commented Feb 13, 2023

AlexLao512 commented Feb 13, 2023

JulianKemmerer commented Feb 13, 2023

JulianKemmerer commented Feb 14, 2023

JulianKemmerer commented Feb 14, 2023

JulianKemmerer commented Feb 14, 2023

JulianKemmerer commented Feb 22, 2023 •

edited

JulianKemmerer commented Feb 23, 2023

AlexLao512 commented Feb 25, 2023

JulianKemmerer commented Apr 26, 2023

JulianKemmerer commented Apr 27, 2023 •

edited

JulianKemmerer commented Apr 27, 2023

JulianKemmerer commented Apr 28, 2023

JulianKemmerer commented Apr 30, 2023

AlexLao512 commented Apr 30, 2023

JulianKemmerer commented May 1, 2023

WIP: CHG: Use pycparser as a git submodule #159

Are you sure you want to change the base?

WIP: CHG: Use pycparser as a git submodule #159

Conversation

AlexLao512 commented Feb 4, 2023

JulianKemmerer commented Feb 5, 2023

ghost commented Feb 5, 2023

AlexLao512 commented Feb 5, 2023 • edited

AlexLao512 commented Feb 5, 2023 • edited

AlexLao512 commented Feb 5, 2023

JulianKemmerer commented Feb 13, 2023

AlexLao512 commented Feb 13, 2023

JulianKemmerer commented Feb 13, 2023

JulianKemmerer commented Feb 14, 2023

JulianKemmerer commented Feb 14, 2023

JulianKemmerer commented Feb 14, 2023

JulianKemmerer commented Feb 22, 2023 • edited

JulianKemmerer commented Feb 23, 2023

AlexLao512 commented Feb 25, 2023

JulianKemmerer commented Apr 26, 2023

JulianKemmerer commented Apr 27, 2023 • edited

JulianKemmerer commented Apr 27, 2023

JulianKemmerer commented Apr 28, 2023

JulianKemmerer commented Apr 30, 2023

AlexLao512 commented Apr 30, 2023

JulianKemmerer commented May 1, 2023

AlexLao512 commented Feb 5, 2023 •

edited

AlexLao512 commented Feb 5, 2023 •

edited

JulianKemmerer commented Feb 22, 2023 •

edited

JulianKemmerer commented Apr 27, 2023 •

edited