Support parsing of v33 data (#47)

* WIP

* WIP CaseID

* WIP Buses

* WIP Loads

* WIP FixedShunts

* WIP Generators

* WIP Branches

* Add some more docs on internals

* WIP Transformers

* WIP Q record

* WIP specify version

* Expand v33 test data

* WIP TwoTerminalDCLines

* Include version in Network

* WIP MultiTerminalDCLines

* More hiding of types in printing

* WIP MultiSectionLineGroups

* WIP FACTSDevices

* WIP SwitchedShunts

* Remove testfile used in development

* Remove GNEDevice placeholder code

* Refactor Branches30/33 to be defined in loop

* Update docs now v30 and v33 both supported

* fixup! Update docs now v30 and v33 both supported

* fixup! Update docs now v30 and v33 both supported

* Add more specific tests for v33 data

* fixup! Update docs now v30 and v33 both supported

* fixup! WIP FixedShunts

README.md

@@ -14,14 +14,11 @@ To read a `.raw` file, use `parse_network`:
 using PowerFlowData
 parse_network("file.raw")
 ```
-This will return a `Network` object, which contains the data parsed into dedicated structures matching the PSS/E-format specification.
+This will return a `Network` object, which contains the data parsed into dedicated structures matching the
+[PSS/E](https://en.wikipedia.org/wiki/Power_system_simulator_for_engineering)-format specification.
 
 **[Documentation](https://nickrobinson251.github.io/PowerFlowData.jl/dev)**
 
-### Stability
-
-Currently this is a work-in-progress package being developed for fun… 'cos you gotta have hobbies.
-It is not stable.
-It is not yet very robust.
-The format specification is based on old [PSS/E](https://en.wikipedia.org/wiki/Power_system_simulator_for_engineering) user-manuals and example files I could find online.
-There are likely more performance improvements to be made too.
+The format specification is based on old PSS/E user-manuals and example files I could find online.
+Currently v30 and v33 of the format are supported.
+Please open an issue if you run into any problems or missing features.

docs/Manifest.toml

@@ -16,9 +16,9 @@ uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
 
 [[Compat]]
 deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
-git-tree-sha1 = "31d0151f5716b655421d9d75b7fa74cc4e744df2"
+git-tree-sha1 = "dce3e3fea680869eaa0b774b2e8343e9ff442313"
 uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
-version = "3.39.0"
+version = "3.40.0"
 
 [[Crayons]]
 git-tree-sha1 = "3f71217b538d7aaee0b69ab47d9b7724ca8afa0d"
@@ -61,15 +61,15 @@ uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"
 
 [[DocStringExtensions]]
 deps = ["LibGit2"]
-git-tree-sha1 = "a32185f5428d3986f47c2ab78b1f216d5e6cc96f"
+git-tree-sha1 = "b19534d1895d702889b219c382a6e18010797f0b"
 uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
-version = "0.8.5"
+version = "0.8.6"
 
 [[Documenter]]
 deps = ["ANSIColoredPrinters", "Base64", "Dates", "DocStringExtensions", "IOCapture", "InteractiveUtils", "JSON", "LibGit2", "Logging", "Markdown", "REPL", "Test", "Unicode"]
-git-tree-sha1 = "8b43e37cfb4f4edc2b6180409acc0cebce7fede8"
+git-tree-sha1 = "f425293f7e0acaf9144de6d731772de156676233"
 uuid = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
-version = "0.27.7"
+version = "0.27.10"
 
 [[Downloads]]
 deps = ["ArgTools", "LibCURL", "NetworkOptions"]
@@ -173,9 +173,9 @@ version = "1.4.1"
 
 [[Parsers]]
 deps = ["Dates"]
-git-tree-sha1 = "a8709b968a1ea6abc2dc1967cb1db6ac9a00dfb6"
+git-tree-sha1 = "ae4bbcadb2906ccc085cf52ac286dc1377dceccc"
 uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
-version = "2.0.5"
+version = "2.1.2"
 
 [[Pkg]]
 deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
@@ -195,9 +195,9 @@ version = "1.0.0-DEV"
 
 [[PrettyTables]]
 deps = ["Crayons", "Formatting", "Markdown", "Reexport", "Tables"]
-git-tree-sha1 = "6330e0c350997f80ed18a9d8d9cb7c7ca4b3a880"
+git-tree-sha1 = "d940010be611ee9d67064fe559edbb305f8cc0eb"
 uuid = "08abe8d2-0d0c-5749-adfa-8a2ac140af0d"
-version = "1.2.0"
+version = "1.2.3"
 
 [[Printf]]
 deps = ["Unicode"]
@@ -255,9 +255,9 @@ version = "1.0.1"
 
 [[Tables]]
 deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "LinearAlgebra", "TableTraits", "Test"]
-git-tree-sha1 = "1162ce4a6c4b7e31e0e6b14486a6986951c73be9"
+git-tree-sha1 = "fed34d0e71b91734bf0a7e10eb1bb05296ddbcd0"
 uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
-version = "1.5.2"
+version = "1.6.0"
 
 [[Tar]]
 deps = ["ArgTools", "SHA"]

docs/make.jl

@@ -16,6 +16,7 @@ makedocs(;
         "Home" => "index.md",
         "API" => "api.md",
         "Alternatives" => "alternatives.md",
+        "Implementation" => "implementation.md",
     ],
     strict=true,
     checkdocs=:exports,

docs/src/alternatives.md

@@ -1,25 +1,29 @@
 # Alternatives
 
-In Julia, I am are aware of two other open-source packages with functionality to parse PSS/E files:
+## Julia
+
+I am are aware of two other open-source packages with functionality to parse PSS/E files:
   - [PowerModels.jl](https://lanl-ansi.github.io/PowerModels.jl/stable/parser/#PTI-Data-Files-(PSS/E))
   - [PowerSystems.jl](https://nrel-siip.github.io/PowerSystems.jl/stable/modeler_guide/generated_parsing/)
 
 I have not used either so cannot recommend one over the other.
 From what I can see, these parsers are almost identical to each other.
 It seems PowerSystems.jl originally vendored the PowerModels.jl code, but the parsers may have diverged slightly over time.
 
-Importantly, these alternatives take a completely different approach to this package.
+Both of these packages only support parsing v33 PSS/E files, and only parse a subset of the data categories in the file.
+Whereas PowerFlowData.jl (this package) can parse all data categories from both v30 and v33 PSS/E files.
+
+Importantly, these alternatives also take a completely different approach to this package.
 These other parsers read the `.raw` files as a `String` (e.g. using `readlines`), then operate on string data, and parse strings into other Julia types as necessary.
+PowerFlowData.jl reads the `.raw` files as a bytes buffer (`Vector{UInt8}`) then parses the bytes directly into Julia types,
+which is much faster and more memory efficient.
 
-PowerFlowData.jl (this package) reads the `.raw` files as a bytes buffer (`Vector{UInt8}`), then parses the bytes directly into Julia types.
+This package was originally developed as a fun exercise, but should correctly implement the full data format specification.
+Please feel encouraged to give this package a try, and open issues if you encounter any
+problems or missing features.
 
-Hopefully this will be much faster and more memory efficient, but benchmarks pending.
-For now, this package is being developed as a fun exercise.
-Feel encouraged to give this package a try (and open issues!), but these alternative parsers are surely more battle-tested!
+## Others
 
-## Implementation details
+In Python, there is a package named [`grg-pssedata`](https://github.com/lanl-ansi/grg-pssedata), which says it parses PSSE v33 data files.
 
-We use [Parsers.jl](https://github.com/JuliaData/Parsers.jl/) to parse bytes into Julia types.
-Broadly speaking, we use `Parsers.Options` to configure the parsing based on the `.raw` format (e.g. `,` characters are delimiters), and then `Parsers.xparse` to actually parse the bytes between delimiters into the expected Julia types.
-The expected Julia type depends on the category of data we are reading at that point in the file (buses, loads, …);
-if the PSS/E user manual says "load records" should come after "bus records", and each load record should have 12 columns with the first column containing an integer "bus number", then we try to parse the first value in a load record as an `Int`, and so on.
+In Matlab, the [`MATPOWER`](https://github.com/MATPOWER/matpower/) package says it has parsing functionality for (unspecified) PSSE data files.

docs/src/api.md

@@ -17,13 +17,20 @@ CaseID
 ## Buses
 ```@docs
 Buses
+Buses30
+Buses33
 ```
 
 ## Loads
 ```@docs
 Loads
 ```
 
+## Fixed Shunts
+```@docs
+FixedShunts
+```
+
 ## Generators
 ```@docs
 Generators
@@ -32,6 +39,8 @@ Generators
 ## Branches
 ```@docs
 Branches
+Branches30
+Branches33
 ```
 
 ## Transformers
@@ -47,6 +56,8 @@ AreaInterchanges
 ## Two-Terminal DC Lines
 ```@docs
 TwoTerminalDCLines
+TwoTerminalDCLines30
+TwoTerminalDCLines33
 ```
 
 ## VSC DC Lines
@@ -57,6 +68,8 @@ VSCDCLines
 ## Switched Shunts
 ```@docs
 SwitchedShunts
+SwitchedShunts30
+SwitchedShunts33
 ```
 
 ## Transformer Impedance Corrections
@@ -69,6 +82,8 @@ ImpedanceCorrections
 MultiTerminalDCLines
 MultiTerminalDCLine
 DCLineID
+DCLineID30
+DCLineID33
 ACConverters
 DCBuses
 DCLinks
@@ -77,6 +92,8 @@ DCLinks
 ## Multi-Section Line Groups
 ```@docs
 MultiSectionLineGroups
+MultiSectionLineGroups30
+MultiSectionLineGroups33
 ```
 
 ## Zones
@@ -97,4 +114,6 @@ Owners
 ## FACTS Devices
 ```@docs
 FACTSDevices
+FACTSDevices30
+FACTSDevices33
 ```

docs/src/implementation.md

@@ -0,0 +1,18 @@
+# Implementation details
+
+This page has rough notes about the internals of PowerFlowData.jl.
+
+We use [Parsers.jl](https://github.com/JuliaData/Parsers.jl/) to parse bytes into Julia types.
+Broadly speaking, we use `Parsers.Options` to configure the parsing based on the `.raw` format (e.g. `,` characters are delimiters), and then `Parsers.xparse` to actually parse the bytes between delimiters into the expected Julia types.
+The expected Julia type depends on the category of data we are reading at that point in the file (buses, loads, …);
+if the PSS/E user manual says "load records" should come after "bus records", and each load record should have 12 columns with the first column containing an integer "bus number", then we try to parse the first value in a load record as an `Int`, and so on.
+
+The aim is to capture the domain knowledge / format specification in the types,
+and keep the parsing code as minimal as possible (relying on the types for the domain knowledge).
+When trying to support multiple versions of the format, the rough strategy is:
+* if a wholly new data category exists in the new format, create a new `Records` subtype
+* if a data category has added a new column to the end of a record, add a new `Union{T, Missing}` field to the existing `Records` subtype
+* if a data category has added/removed columns in the middle of a record, or changed the element type of a column, create a new `Records` subtype with a common supertype as the existing `Records` subtype for the category
+  (e.g. `Buses30 <: Buses`, `Buses33 <: Buses`, `Buses <: Records`)
+* if a version-specific `Records` subtype is being used, update the top-level `parse_network` function to parse the records into the appropriate type, and in the order expected by the version of the format
+  (using the version number extracted as part of the Case ID data).

docs/src/index.md

@@ -13,6 +13,14 @@ parse_network("file.raw")
 ```
 This will return a [`Network`](@ref) object, which contains the data parsed into dedicated structures matching the PSS/E-format specification.
 
+You can specify what version of the PSS/E data format the file is in using the `v` keyword,
+like:
+```julia
+parse_network("file.raw"; v=33)
+```
+Alternatively, the version of the data format will automatically be determined when parsing.
+Versions 30 and 33 of the format are currently supported.
+
 ## Example
 
 Usually your data will be in a file, and you'd read it with `parse_network("file.raw")`,
@@ -21,15 +29,15 @@ but here we'll pass in the data directly, to show how it matches to the output:
 ```@repl
 using PowerFlowData, DataFrames
 data = IOBuffer("""
-    0,   100.00          / PSS/E-29.3    WED, SEP 15 2021  21:04
+    0,   100.00          / PSS/E-30.3    WED, SEP 15 2021  21:04
     SE SNAPSHOT 15-09-2021 PEAK CASE 18:00
     FULL COPY OF SYNTHETIC
          1,'AAA    3    ', 111.0000,4,     0.000,     0.000, 327,   1,0.00000,   0.0000,   1
     222222,'PRPR C D    ',  42.0000,1,     0.000,     0.000, 694,  24,1.11117,  20.0606,   7
     0 / END OF BUS DATA, BEGIN LOAD DATA
     """
 );
-network = parse_network(data);
+network = parse_network(data; v=30);
 NamedTuple(network.caseid)  # Case Identification data is a single row.
 DataFrame(network.buses)    # Bus data, and all other data, is a table.
 ```

src/PowerFlowData.jl

@@ -9,10 +9,16 @@ using Tables
 
 export parse_network
 export Network
-export CaseID, Buses, Loads, Generators, Branches, Transformers, AreaInterchanges
-export TwoTerminalDCLines, VSCDCLines, SwitchedShunts, ImpedanceCorrections
-export MultiTerminalDCLines, MultiTerminalDCLine, DCLineID, ACConverters, DCBuses, DCLinks
-export MultiSectionLineGroups, Zones, InterAreaTransfers, Owners, FACTSDevices
+export CaseID, Buses, Buses30, Buses33, Branches, Branches30, Branches33
+export FixedShunts, Loads, Generators, Transformers, AreaInterchanges
+export SwitchedShunts, SwitchedShunts30, SwitchedShunts33
+export TwoTerminalDCLines, TwoTerminalDCLines30, TwoTerminalDCLines33
+export VSCDCLines, ImpedanceCorrections
+export MultiTerminalDCLines, MultiTerminalDCLine, DCLineID, DCLineID30, DCLineID33
+export ACConverters, DCBuses, DCLinks
+export MultiSectionLineGroups, MultiSectionLineGroups30, MultiSectionLineGroups33
+export Zones, InterAreaTransfers, Owners
+export FACTSDevices, FACTSDevices30, FACTSDevices33
 
 include("debug.jl")
 include("types.jl")