Current status of parsing and plc-autodoc

[RobertoRoos]

Hi @klauer ,

We talked in pcdshub/lcls-twincat-general#80 (comment) where I learned about this project. I started my own Sphinx-PLC integration a while back and the similarities are striking.

What is the current status of your ST parsing? I am only interested in the Declarations. Are comments preserved?

My parser (based on TextX) seems to work for all Structured Text declarations now, but it's not exactly robust. I might go over to yours instead. I guess in general I am interesting in putting (parts of) these projects together.

[klauer]

Thanks for reaching out, @RobertoRoos! There are definitely similarities in our projects which is neat. TextX is new to me, but it looks like a pretty powerful library to develop on top of.

blark is a fun side project of sorts for me, so it probably can't compete in many ways. I wouldn't be offended (of course!) if you decided to continue on your current path - and would be interested in seeing how it works out for you. I should add your project to the 'related projects' in the README section at some point...

What is the current status of your ST parsing?

Some random points about the status/goals of blark:

1. My primary aim is to have it successfully parse any and all valid structured text code that TwinCAT thinks is syntactically correct. At the moment, I'm not aware of any valid source code that it fails to parse - but don't take that as "blark is perfect!" rather "blark needs more test cases".
2. For example, there have been holes in the grammar that we've discovered and have been patching as needed (see the recently closed issues)
3. A lot of these changes aren't in the latest release but soon should be. Anything I mention here really applies to the current master branch.
4. With the solution parser from a recent PR, I'm also tweaking how blark gets its source code - there's now a built-in solution/project loader that extracts structured text from POUs and allows for re-writing/reformatting/etc.
5. There are some upcoming language features that may need grammar changes (see Language features in upcoming TwinCAT 4026 release #71 )
6. I can't guarantee stability as the project is used by 2 people at this point - but I'd be happy to work with you on feature/API changes/request/etc

I am only interested in the Declarations.

You can selectively parse sections of code - just declaration or just implementation.

Are comments preserved?

Yes, comments are mostly preserved (* if you work within the bounds of the blark transformed dataclasses at least; if you want to use the underlying grammar parser without the machinery that blark adds, then it'll be extra work on your end)

The blark.transform layer wraps up comments in the meta attribute, and (if desirable) you can use the blark.summary layer for a higher-level view of the code.

The test suite shows this off a bit - reading in declarations and verifying that it matches up comments appropriately:
These checks

blark/blark/tests/test_summary.py

Lines 201 to 244 in 6fae121

	check_declarations(
	system.declarations,
	[
	DeclarationCheck(
	name="xSwAlmRst",
	base_type="BOOL",
	comments=["(* Global Alarm Reset - EPICS Command *)"],
	),
	DeclarationCheck(
	name="xAtVacuum",
	base_type="BOOL",
	comments=["(* System At Vacuum *)"],
	),
	DeclarationCheck(
	name="xFirstScan",
	base_type="BOOL",
	comments=[
	"(* This boolean is true for the first scan, and is false "
	"thereafter, use for initialization of stuff *)",
	],
	),
	DeclarationCheck(
	name="xOverrideMode",
	base_type="BOOL",
	comments=[
	"//This bit is set when using the override features of the system"
	],
	),
	DeclarationCheck(
	name="xIOState",
	base_type="BOOL",
	comments=["(* ECat Bus Health *)"],
	),
	DeclarationCheck(
	name="I_EcatMaster1",
	base_type="AMSNETID",
	location="input",
	comments=[
	"{attribute 'naming' := 'omit'}",
	"(* AMS Net ID used for FB_EcatDiag, among others *)"
	],
	),
	],
	)

Correspond to this code https://github.com/klauer/blark-twincat-root-example/blob/16191a36164ff215414a1b67f2d6f95eb5bd207b/lcls-twincat-general/v2.8.1/LCLSGeneral/LCLSGeneral/Data%20types/Misc/ST_System.TcDUT#L8-L14

Now that's a bit complicated, code-wise to dig through.A bit more tersely (this is running on blark master keep in mind!):

import blark.parse

# Pick the same DUT from the test suite
fn = "blark/tests/twincat_root/lcls-twincat-general/v2.8.1/LCLSGeneral/LCLSGeneral/Data types/Misc/ST_System.TcDUT"
(parsed,) = list(blark.parse.parse(fn))

# Transform the lark tokens and rules into blark dataclasses:
transformed = parsed.transform()

# Pick off the meta information from the first declaration:
meta = transformed.items[0].declaration.declarations[0].meta
print("Comments for the first declaration:", meta.comments)

$ python discussion_80.py
Comments for the first declaration: [Token('MULTI_LINE_COMMENT', '(* Global Alarm Reset - EPICS Command *)')]

Now if you were OK working with the summary layer, this gets a bit easier. This is where I'm initially handling all the sphinx domain stuff:

import blark.summary
summary = blark.summary.CodeSummary.from_parse_results([parsed])

comments = summary.data_types["ST_System"].declarations["xSwAlmRst"].meta.comments
print("From the summary", comments)

The existing sphinx domain code may be broken as-is but I plan to get back to it at some point. My team was less interested in PLC API documentation in Sphinx than I had hoped! (Code is here: https://github.com/klauer/blark/blob/master/blark/sphinxdomain.py )
We instead have just code summaries on our project pages: https://pcdshub.github.io/lcls-twincat-general/v2.8.2/ (for example)

Hope that clarifies things a bit.

[RobertoRoos]

Thanks for the overview! blark seems much stronger in ST parsing than what I'd made up, I'll see if I can at least incorporate it directly.
I'll keep going with my thingy until a minimum working version to see where it'll end up.

My team was less interested in PLC API documentation in Sphinx than I had hoped!

Lol, ain't that the truth. It's good to see you using Sphinx at all, my colleagues largely insist on separate Word documents...