Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v0.12.x mega-merge #109

Merged
merged 27 commits into from
May 2, 2022
Merged

v0.12.x mega-merge #109

merged 27 commits into from
May 2, 2022

Conversation

PMeira
Copy link
Member

@PMeira PMeira commented Feb 27, 2022

This is a squashed version of 270+ commits that merge a variety of features. I'll still try to group the commits in a few groups instead, but the priority now is to test all downstream projects.

As usual, this should be up-to-date with the official OpenDSS SVN, r3366.

A lot of important changes (will be updated as required), besides the long list already in https://github.com/dss-extensions/dss_capi/blob/master/docs/changelog.md#version-012

  • The whole property system and a lot of the parsing functions were rewritten allowing new features and less code duplication. This will also help a lot when we migrate from Pascal. This is a major change. We no longer keep copies of every property as strings, for example. Most of the code now uses scoped enums to refer to properties (remaining uses of magic numbers should be addressed for v1.0).
  • Usage of the text parser was reduced in most of the API functions. Before, a lot of the modules created an edit string and passed that to the parser, defeating some of the purpose of the specific APIs. Now, the new property system is fed directly, eliminating the cost of the string formatting and then parsing.
  • Unused code and a lot of redundant code was removed, hence almost 23k lines removed, even though new features were added.
  • New Obj_* and Batch_* sets of functions for the C API: these do less redundant validation and allows more direct manipulation of objects. These open many possibilities. A new header that includes the property indices was added, more tools and artifacts to automate bindings can be still be added (e.g. type of the properties). A new function DSS_ExtractSchema returns the basic structure of all classes (JSON encoded string), including indices, internal types (subject to change for future releases), and related enumerated types -- this is intended for documentation and automation tools. Closes Create missing APIs #11
  • ZIP file support for input scripts.
  • Some utility functions to load multiple commands at once (either from one single big string or multiple strings).
  • Error messages now include trace-back.
  • For i18n, to allow easier maintenance, to decouple documentation from the code, all property/command/option descriptions were removed from the code. A new repo will be published soon, which will contain the old strings (slightly modified version of original English strings, and a Brazilian Portuguese version), instructions and so on. Most users don't use help functions so this is not a deal breaker for testing. The implementation uses gettext catalogs but they're not required to run. Related: String and help localization dss-extensions#4
  • DSSContexts were tested and stabilized. Most of the global variables were encapsulated in DSSContexts before but other issues remained and were addressed here. DSSContexts can be used in user-controlled threads instead of the internal actors, so a lot more flexible for advanced users.
  • The parallel-machine implementation based on our "DSSContexts" was finished and tests on Windows and Linux. This is a difference implementation from the official code and the one we used to have in 2018. Some thread synchronization features were used in a few critical points to ensure some thread safety with the internal actor mechanism. Diakoptics is still disabled (see notes on v8+ to v7: porting remaining features  #46). These should address a bunch of tickets such as dss._cffi_api_util.DSSException DSS-Python#40, Thread-safety for parallel calculations and reading results dss_sharp#10, Implementing Parallelization with OpenDSSDirect same as OpenDSS Version 8 OpenDSSDirect.py#90, several emailed requests, etc.
  • Complex numbers were migrated to used FPC's uComplex as the base structure, which support operator overloading and easier mixed-type operations (e.g. scale a complex by float64, etc.)
  • Fix dangerous memory issues (invalid accesses) and memory leaks.
  • Adjust output filenames in general, using lowercase file extensions, circuit/case prefix, etc., partially matching the behavior from the upstream OpenDSS.
  • Some minor optimizations. They add up, and some circuits can run quite a bit faster than v0.10.7.
  • A few unused properties were removed to avoid misleading users. Since DSS C-API is not often used interactively, it's better to error when loading a file to indicate potential issues than leave a user wondering why changing certain property does not change anything.

Some other details:

  • The GISCoords command was added (as a no-op command) to help IEEE 13-bus (extended) Conversion Errors NREL/ditto#387 (note that the sample system there has issues in case-sensitive file systems). As a reminder, OpenDSS-GIS itself is not supported as we wouldn't be able to test it (besides using TCP connections).
  • Fixed issues with energy meter reports (reported on the forum)
  • Added Set DefaultBaseFreq=... on Save Circuit.
  • UTF8 support: the basics (file paths, strings through the C API, and input/output to files) work, needs some more testing in varied scenarios. Please report if you encounter issues. Closes Revisit string encoding #6

Remaining tasks:

  • (in progress) Documentation of the new features and features with different implementations on DSS C-API vs. EPRI's OpenDSS binaries.
  • Basic updates to OpenDSSDirect.jl (no new API yet)
  • Basic updates to OpenDSSDirect.py (no new context API yet, pending Class iterators OpenDSSDirect.py#78 ) -- pending PR since we also need a public push of DSS Python, see below
  • Update the PR for DSS Sharp: Extensive updates to support multiple DSS instances, GR API, and exception mapping. dss_sharp#11
  • Reports and dumps: The general results have been validated but a bunch a stuff needs to be tested somewhat manually still, like reports and dumps. Update: not every single option was retested but the general status is good.
  • Finish the initial proof-of-concept of usage of the API features on DSS Python. Since this is a new development, we can share it completely with OpenDSSDirect.py. This should handle Try to convert dss.Properties.Value() return to numbers OpenDSSDirect.py#104, Rework DSSProperty OpenDSSDirect.py#21, part of Common interface for exporting data OpenDSSDirect.py#74 and other issues.
  • Finish the API for the (three?) types of plots that still need some work. Plotting and some other features now have optional callbacks. An implementation of most plots from DSSView.exe (the tool that comes with OpenDSS) and OpenDSS Viewer (optional) is already in-progress and will be merged to DSS Python.
  • Now that we have a PM implementation, update the user-model code and headers accordingly. Not many people use these at all, but it's an easy task. Updated but all these will need more work (not targeted for this release)

If any of the new API features needs too much work, we can release a v0.12.0 with those disabled/unsupported and enable them in another release so this doesn't wait too much longer.

This should be one of the last big releases before v1.0, and v1.0 will be last version based on Pascal. We hope to release at least a preview of v2.0 in the coming months. The list above is essential to allow an easier migration.

…t that's being compiled with an external call"

(note: no code was reused for this one)
git-svn-id: https://svn.code.sf.net/p/electricdss/code/trunk@3414 d8739450-1e93-4ef4-a0af-c327d92816ff
- reset error state when running ZIP_Redirect
- try exchanging between forward slashes and back slashes
The functions List, Constains and Extract complement basic usage of ZIP files when the users include other files in the ZIP and/or employ some sort of logic to filter/discover the files in the ZIP to run the circuits inside it.
@PMeira PMeira marked this pull request as ready for review May 2, 2022 21:50
@PMeira
Copy link
Member Author

PMeira commented May 2, 2022

I'm merging this into master to avoid new developments based on the obsolete 0.10.x. I'll continue working on the finishing touches there or on incremental PRs.

@PMeira PMeira merged commit 0d8d630 into master May 2, 2022
@alourie
Copy link

alourie commented May 5, 2022

Amazing work, @PMeira! It was definitely the time to get it merged and work on the future updates from the master.

@PMeira PMeira deleted the 0.12.x branch July 12, 2022 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create missing APIs Revisit string encoding
2 participants