Strip param inlining #325
Merged
Strip param inlining #325
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closed
MatthewSteen
added a commit
that referenced
this pull request
Sep 20, 2024
* Fix CLI (#290) * Fix docker (#288) * Squish (#289) * Exclude python 3.12 from support (#293) * exclude python 3.12 from support * change pygit2 version * move cd.yaml to poetry * prevent werkzeug from breaching 3.0.0 * update checkout and setup-python. Add 3.11 to test matrix * Refactor Table Connection (#291) * unify getting objects by id and consolidate to delete language * fix tests * Documenting shape-to-template process and adding some QoL improvements (#278) * documenting shape-to-template process and adding some QoL improvements * fix test * skip 223P tests until gabe can fix them after next standard prerelease is done * remember to add shapes from the graph even when loading from directory * add exception to catch lack of sh:path * add test cases for recent code changes * dedicated test suite for library shape test * dedicated test suite for library shape test * Add Application Explorer (#292) * Add Application Explorer * Address comment * Fix test * improve performance by not requiring OPTIONAL clause in SPARQL queries --------- Co-authored-by: Gabe Fierro <gtfierro@mines.edu> * Generate SPARQL Queries from SHACL shapes (#273) * add method to generate sparql query from shacl shape * add some tests * fix typo in query generation * add sh:node, sh:or * fixup tests * handle or inside qualifiedvalue shape, complex property paths * add test file * update docs with image and guide on query generation * augment docstring with more features * add test case * add section on supported SHACL features * add algorithm documentation * support other target definitions * add more tests * Fixing template inlining (#298) * address ambiguous template naming in inline_dependencies * fill non-graphs, propagate the optional arg requirement * update tests * adjusting tests for inlining * Allow Library to find template definitions from dependent libraries (#307) * allow Library to find template definitions from dependent libraries * fixing tests to catch warnings, test dependency tracking * use non-deprecated warnings; make sure warnings are emitted when necessary * clarify when dependencies are infered for SHACL shapes -> templates * update dependencies * adding documentation on templates * placeholder for further template docs * clarify load order * update jsonschema, skip mypy checking on imports * fix API response and test to handle Brick * cleaning some code up * Add ability to use TopQuadrant SHACL implementation as engine (#308) * add shacl_validate/infer functions and use these as the entrypoint. Augment tests to check both shacl engines * fix interactions with shacl inference * tightening up the implementation and use of the shacl_* methods * support specifying shacl engine in the API * update tests; test both pyshacl and topquadrant * add brick-tq-shacl dep * add TODOs * Formatting * no more 3.8! * ignoring some imported packages without type annotations * more type annotations * add types, ignore type errors for imports * update mypy, fix some issues and ignore some others * fix union type annotation * update docker containers * 3.8.1 python for higher * add back python 3.8 * change 3.8 version * add test for finding reasons with a given severity * update brick-tq-shacl, fix type signature * remove debug serializations * bump shacl version * fixing skolemization for validation * move shacl engine config inside buildingmotif object * install rtd deps * jupyter-book needs to be installed pre-build * cleanup and update to default config comments * change from ubuntu:latest to python:3.9 for bacnet testing (#317) * Fix tutorials (#310) * updating validation tutorial * fixing some tutorials * updates * more components * fix imports and formatting and type annotations * fix formatting * fix merge * fix bacnet docker * fixing versions in docker bacnet setup * alphabetize prefixes * fixing model_correction file * Add new transitive_parameters method to accelerate library loading (#314) * Add new transitive_parameters method to accelerate library loading The prior implementation of check_template_dependency_relationship used inline_dependencies to check the parameters of template dependencies. inline_dependencies can be slow-ish because it copies the bodies of templates (these are graphs, so they inherit the performance characteristics of the underlying store). We can compute the parameters through the dependency tree automatically without copying the graphs, which is much faster. * use cached_property * remove cached_property * fix docker * fix bacnet docker container * bump workflow action versions * BMS naming convention parsing (#286) * adding label parsing initial implementation with tests * fix bug in many parsing * test first_true * add ingress and example * adding/testing maybe parser * add errors * propagate errors * update notebook * fix test import * capture errors in the token stream * fix notebook kernel * some changes * Make Parsers classes * Add Application Explorer (#292) * Add Application Explorer * Address comment * Fix test * improve performance by not requiring OPTIONAL clause in SPARQL queries --------- Co-authored-by: Gabe Fierro <gtfierro@mines.edu> * Generate SPARQL Queries from SHACL shapes (#273) * add method to generate sparql query from shacl shape * add some tests * fix typo in query generation * add sh:node, sh:or * fixup tests * handle or inside qualifiedvalue shape, complex property paths * add test file * update docs with image and guide on query generation * augment docstring with more features * add test case * add section on supported SHACL features * add algorithm documentation * support other target definitions * add more tests * Fixing template inlining (#298) * address ambiguous template naming in inline_dependencies * fill non-graphs, propagate the optional arg requirement * update tests * adjusting tests for inlining * Allow Library to find template definitions from dependent libraries (#307) * allow Library to find template definitions from dependent libraries * fixing tests to catch warnings, test dependency tracking * use non-deprecated warnings; make sure warnings are emitted when necessary * clarify when dependencies are infered for SHACL shapes -> templates * update dependencies * adding documentation on templates * placeholder for further template docs * clarify load order * update jsonschema, skip mypy checking on imports * fix API response and test to handle Brick * cleaning some code up * remove unused import, fix type signature * fix poetry lock * moving tokens/parser to different files * adding combinators module * add some negative cases * add missing combinators file * fix notebook --------- Co-authored-by: Hannah Eslinger <heslinge@nrel.gov> * New model builder (#301) * address ambiguous template naming in inline_dependencies * fill non-graphs, propagate the optional arg requirement * update tests * adjusting tests for inlining * update 223p templates * update 223 * add model builder impl * add QUDT * update templates * allow adding triples directly to the builder graph * allow Library to find template definitions from dependent libraries * fixing tests to catch warnings, test dependency tracking * use non-deprecated warnings; make sure warnings are emitted when necessary * clarify when dependencies are infered for SHACL shapes -> templates * update dependencies * adding documentation on templates * placeholder for further template docs * clarify load order * update jsonschema, skip mypy checking on imports * fix API response and test to handle Brick * updating 223p and templates * filtering validationcontext by severity * add shacl_validate/infer functions and use these as the entrypoint. Augment tests to check both shacl engines * fix interactions with shacl inference * tightening up the implementation and use of the shacl_* methods * support specifying shacl engine in the API * update tests; test both pyshacl and topquadrant * add brick-tq-shacl dep * add TODOs * Formatting * no more 3.8! * ignoring some imported packages without type annotations * more type annotations * add types, ignore type errors for imports * update mypy, fix some issues and ignore some others * fix union type annotation * update docker containers * 3.8.1 python for higher * add back python 3.8 * change 3.8 version * add test for finding reasons with a given severity * update brick-tq-shacl, fix type signature * remove debug serializations * bump shacl version * fixing skolemization for validation * update 223p, fix merge error * fix notebook * fix the merge * remove qudt from commit * remove more bad merge * fix more bad merge * remove more bad merge * add s223 namespace, parameter prop on template builder * add model builder draft * add call syntax to simplify evaluating templates * do not waste cycles re-binding namespaces on copied graphs * update model builder notebook * add java for topquadrant support * use topquadrant in 223 notebook * updating dockerfile; this started failing, possibly because of updated ubuntu * properly initialize graph superclass * Add graph hashing method to utils (#296) * Add graph hashing method to utils * change to approximate hash and use existing model for test * make test use a model graph to show positive functionality with DB stored graphs * use new rdflib triple canonicalizer to calculate graph hashes * update docstring * rename from approximate_graph_hash to graph_hash * Create generic serializer deserializer for parsers (#322) * Strip param inlining (#325) * add _strip_param function and tests to support inlining * fix test * Changes to ingresses to support external work (#312) * Allow TemplateHandler to handle edge cases in template eval Adds two flags: - require_optional - fill_unused These allow adjustment of how the template ingress handler deal with records that don't necessarily have all the parameters for the chosen template (fill_unused), and how the handler deal with optional parameters in the chosen template (require_optional) * more configuration on template ingress * put limit on xlsx * cleaning up parameter,s adding docstrings * Update template.py * Update xlsx.py * use pathlike * Fix autogeneration of templates (#329) * factor out the guarantee method and add unit tests * add function to utils * Update 223P to public advisory review version (#309) * updating 223p again * update 223p templates * update 223 * add QUDT * update templates * allow Library to find template definitions from dependent libraries * fixing tests to catch warnings, test dependency tracking * use non-deprecated warnings; make sure warnings are emitted when necessary * clarify when dependencies are infered for SHACL shapes -> templates * update dependencies * adding documentation on templates * placeholder for further template docs * clarify load order * update jsonschema, skip mypy checking on imports * fix API response and test to handle Brick * updating 223p and templates * filtering validationcontext by severity * add shacl_validate/infer functions and use these as the entrypoint. Augment tests to check both shacl engines * fix interactions with shacl inference * tightening up the implementation and use of the shacl_* methods * support specifying shacl engine in the API * update tests; test both pyshacl and topquadrant * add brick-tq-shacl dep * add TODOs * Formatting * no more 3.8! * ignoring some imported packages without type annotations * more type annotations * add types, ignore type errors for imports * update mypy, fix some issues and ignore some others * fix union type annotation * update docker containers * 3.8.1 python for higher * add back python 3.8 * change 3.8 version * add test for finding reasons with a given severity * update brick-tq-shacl, fix type signature * remove debug serializations * bump shacl version * fixing skolemization for validation * update 223p, fix merge error * fix notebook * fix more templates; test 223p again * move shacl engine config inside buildingmotif object * parameterize testing of templates for 223p * change some brick templates to make tests less noisy * fixing up templates * add brick integration tests * upadte QUDT * 223p fixup * add shacl engine and qudt to tests * use public review version of 223p * add qudt 2.1.37 release * better exception message * fixing system, sensor templates * fixing up 223 templates * fix plugging connection points in 223p * add the shape graph to the data graph for validation * clean up a little; skip some templates for pyshacl + 223p * fixing use of tmpdir -> tmp_path * try persistence test without path * use path again; try print statements * Add new transitive_parameters method to accelerate library loading The prior implementation of check_template_dependency_relationship used inline_dependencies to check the parameters of template dependencies. inline_dependencies can be slow-ish because it copies the bodies of templates (these are graphs, so they inherit the performance characteristics of the underlying store). We can compute the parameters through the dependency tree automatically without copying the graphs, which is much faster. * use cached_property * tryin-memory? * skip the test -- is this the only one that iss broken? * remove this file * add back path, remove other conftest * add back path, remove other conftest * remove test from in-progress code * remove cached_property * add back conf tests with optimized loading * fix docker * more test hacking * remove duplicate parameterization * remove bad merge * fix bacnet docker container * bump workflow action versions * use more recent Brick, try to write to local db file for persistence test * fix brick references, fix persistence test * support configuring the shacl engine when starting the buildingmotif API * make sure SHACL engine is passed around * handle dependencies for Brick validation in integration tests * bump deps * remove debug * add imports for brick * update brick 1.3 -> 1.4 * reduce tests temporarily * fixing flags to customize how much work gets done when a library is loaded * build templates off of imports closure * more recent 223p build * update again * newer 223p * temporarily pause bacnet integration * add custom option; this needs to be done at the "top level" of the pytest config * add bacnet back * fix ci configuration * fix dependencies on template * fixing some libraries * add _strip_param function and tests to support inlining * fix last class * fix test * add _strip_param function and tests to support inlining * fix test * revert db persistence test * change how we indicate buildingMOTIF's singleton metaclass * update chiller pointlist library * add new test subdirectory to fucus on library/template tests * remove debugs * monkeypatching to handle the singleton * more adjustments on the monkeypatching * more monkeypatching * add comments for explaining the library conftest * add library tests for github, add back notebook fixture * add IDs to notebook tests * factor out the guarantee method and add unit tests * add function to utils * update internal Brick * update notebooks to match buildingmotif API * fixing a couple notebooks * fix kernel * skip some 223p templates that cannot stand on their own * more templates to skip for 223p * increase notebook run time * remove unneeded flag * small changes to template compilation * fix chiller templates * add libraries to dockerfile so that pytest can enumerate the tests * add docstring for shacl_engine * cleaner bypass of the singleton for library tests --------- Co-authored-by: Matt Steen <MatthewSteen@users.noreply.github.com> * Create parser UI (#319) * Add parsers endpoint * Update node version * Create Parser UIs * Fix linting * Fix linting? * Add in Blockly * WIP * Fix * use inspect to help generate argument serialization * Clean up * Fix spilt --------- Co-authored-by: TShapinsky <tobiasshapinsky@gmail.com> * rename 5.16 * cleanup 5.16 prefixes * add 5.16 missing fault conditions 1 and 9-15 * cleanup 5.16 fault conditions 2-8 * typos * add 5.17 fault conditions * add 5.18 fault conditions * remove unused components prefix * add 5.22 fault conditions * update Operating State points * changing the model of outside air fraction * remove internal variables (constants) * Updating setuptool to use the newest version, bumping other deps too (#340) * Updating setuptool to use the newest version, bumping other deps too * update pyshacl, other deps * Update pyproject.toml * update poetry.lock * Update NREL 223P Templates (#343) * add new pipe and belimo energy valve templates, update property definitions for water and air mediums, remove QUDT files not needed for model validation * fix dependency * skip pipe template due to connections * add %OAmin * remove %OA * update test_model.py * fix typo in FCU prefix * Skip ipynb checkpoint directory (#330) * skip .ipynb_checkpoints when scanning for files * do the same filter in utils * add ipynb checkpoint test * force-add the ipynb checkpoitn file * Service clean up (#335) Co-authored-by: Gabe Fierro <gtfierro@mines.edu> * Add test and fix for #326 (#327) * add test and fix for #237 * add negative test * fix tests for windows (#318) * fix tests for windows * flush buffer in tests * Add explanation of point label parsing (#344) * add explanation of point label parsing * add example from notebook * add note on error handling * add links to a few classes * Update docs/explanations/point-label-parsing.md Co-authored-by: Matt Steen <MatthewSteen@users.noreply.github.com> * Update docs/explanations/point-label-parsing.md Co-authored-by: Matt Steen <MatthewSteen@users.noreply.github.com> * Update docs/explanations/point-label-parsing.md Co-authored-by: Matt Steen <MatthewSteen@users.noreply.github.com> * Update docs/explanations/point-label-parsing.md Co-authored-by: Matt Steen <MatthewSteen@users.noreply.github.com> * Update docs/explanations/point-label-parsing.md Co-authored-by: Matt Steen <MatthewSteen@users.noreply.github.com> --------- Co-authored-by: Matt Steen <MatthewSteen@users.noreply.github.com> * Fixing runtime issues in Jupyter notebook (#348) * move the target query out of the hot loop; we probably do not need to compute the cbd because we are already passing in the ontology graph * reduce the number of shapes we check in the notebook * fix composition of shacl shapes * break out integration tests to only run once * remove commented line * break out libraries into its own test * remove libraries from integration * try to run library tests on approval/merge * use new model * try more triggers * make only library tests run when event type is not push * remove style dependency for library tests --------- Co-authored-by: TShapinsky <tobiasshapinsky@gmail.com> * Update Brick to 1.4.1 (#346) * update Brick to 1.4.1 * Update Brick 1.3 -> 1.4 references * move the target query out of the hot loop; we probably do not need to compute the cbd because we are already passing in the ontology graph * reduce the number of shapes we check in the notebook * fix composition of shacl shapes * break out integration tests to only run once * remove commented line * break out libraries into its own test * remove libraries from integration * updating unit tests for brick 1.4 * adjust test to use brick 1.4 * using Brick 1.4 for more tests * fix brick uri * fix 4.6 shape for library tests * Add parser vis (#334) * update pyproject.toml for release --------- Co-authored-by: Hannah Eslinger <heslinge@nrel.gov> Co-authored-by: Tobias Shapinsky <tobiasshapinsky@gmail.com> Co-authored-by: Gabe Fierro <gtfierro@mines.edu> Co-authored-by: daniel <78173152+dllliu@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Required for #309 . When we recursively inline parameters, the names were not normalized and so subsequent evaluation of inlined templates did not work properly.