Access UK renewable data using Go
This is based on my pywind library but doesn't yet offer the same functionality. Presently it is very verbose and some of the exporting isn't ideal.
Presently there are a few things working :-)
gore $ cd cmd/gore
gore/cmd/gore $ go build
gore/cmd/gore $ ./gore.exe -help
UK Renewables App
=================
Usage of ./gore:
./gore command [parameters] [flags]
Available Commands
==================
b1320 - Elexon: B1320
Congestion Management Measures: Countertrading
b1330 - Elexon: B1330
Congestion Management Measures: Costs of Congestion Management
b1420 - Elexon: B1420
Installed Generation Capacity per Unit
b1610 - Elexon: B1610
Actual Generation Output per Generation Unit
b1630 - Elexon: B1630
Actual Or Estimated Wind and Solar Power Generation
certificatesearch - Ofgem Certificate Search
Ofgem: Search certificate database
derbmdata - Elexon: DERBMDATA
Derived BM Unit Data - Multiple Result Sets
dersysdata - Elexon: DERSYSDATA
Derived System Wide Data
fuelinst - Elexon: FUELINST
Generation by Fuel Type (24H Instant Data)
stationsearch - Ofgem Station Search
Ofgem: Search the station database
Options
=======
Options available for all commands:
-exportfilename string
Filename for exported data
-exportformat string
Export format [json, xml, csv]
-log string
Log filename to write to (default "gore.log")
-name string
Name to search for
-v Verbose output (disables logging to a file)
Options available for Ofgem commands:
-month int
Specify a month (default -1)
-scheme string
Ofgem Scheme (RO, REGO)
-year int
Specify a year (default -1)
Options available for Elexon commands:
-bmunit string
BMUnit to search for (Elexon or Ofgem)
-date string
Date to process for (format is YYYY-MM-DD) (defaults to yesterday) (default "2022-11-28")
-elexonkey string
Elexon API Key (required for all Elexon commands) (default "elexon.key")
-period int
Settlement Period for Elexon (1-50) (default -1)Some of it even works :-) For example,
$ gore\cmd\gore> .\gore.exe -elexonkey ..\..\elexon.key -date 2022-02-01 -settlementperiod 30 b1320
UK Renewables App
=================
Getting data for Elexon Report B1320 [ Congestion Management Measures: Countertrading ]...
B1320 API call succeeded, but no data returned$ gore\cmd\gore> ./gore -elexonkey ..\..\elexon.key -exportformat json -exportfilename "test.json" fuelinst
UK Renewables App
=================
Getting data for Elexon Report FUELINST [ Generation by Fuel Type (24H Instant Data) ]...
Query succeeded. 283 items returned
Date Time Period Biomass CCGT Oil Coal Nuclear Wind PS NPSHYD OCGT Other IntFR IntIRL IntNED IntEW IntNEM IntElec IntIFA IntNSL
========== ===== ====== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ========
2022-04-18 21:45 46 2165 15280 0 0 4955 3458 0 229 4 263 0 0 0 0 0 0 0 0
2022-04-18 21:50 46 2164 15106 0 0 4921 3428 0 229 4 266 0 0 0 0 0 0 0 0
2022-04-18 21:55 46 2163 15067 0 0 4886 3405 0 229 4 249 0 0 0 0 0 0 0 0
2022-04-18 22:00 46 2163 15055 0 0 4860 3375 0 218 5 244 0 0 0 0 0 0 0 0
...
Exporting data to test.json as json
Export completedneed to add ability to set dropdown valuesadd elexon api options from pywindimprove the export formattingadd a full command line appreduce the verbosity and improve the logging- add testing
- add docs on how to use command line app
I've reworked the gore app to accept the command as the first argument and then parse the arguments following. It's a bit of a hack to get the arguments processed, but it works and makes the code easier to maintain. Also, this feels more obvious when picking this up after a long absence (ahem). Given this changes usage the version will move to 0.2.0-alpha.
I also improved the error logging to show the error in one case where it wasn't obvious what was wrong. Hotel WiFi has a lot to answer for.
This code has always been covered by the Unlicence, but I hadn't included it in the repository. Now I have :-)
Some Ofgem records have a nil entry for the Accreditation Date. Catch this and return a default value of today.
Fix a silly bug in the command line app and correct some syntax in the README file.
Catch NULL for a number and return a suitable zero value when converting to a number. Additionally remove newlines from key file as these prevent the key from being recognised.
The period string returned in certificate search results can be the simpler "Feb-2022" or the rather less helpful "01/02/2022 - 28/02/2022". Small change to fix this and ensure the "Feb-2022" format is returned in results.
Found some issues with displaying Ofgem Station searches and so have adjusted the output to correct them. This led to realisation the date wasn't correct for the Certificate Search so that's also been updated :-) Along the way I rationalised the flags to remove the period as the year and month provide a better intreface. This allows settlementeriod to become simply period, which should make things simpler to use. Added the DERSYSDATA report from Elexon. Version 0.1.3-alpha pushed to capture these changes.
Tidied up the cmd app and actually started making it useful. Exporting to XML and JSON should now work, though not for the multiple return reports (presently only DERBMDATA) as I'm still trying to get my head around how to do that in a way that makes sense. There is a lot of potential for tidying up yet, but as Justin used to remind everyone, "premature optimisation is evil" so things are just as they are. The data that is returned seems reasonable and should be simple enough to understand. It's faster than pywind for all operations and while it doesn't quite have the same functionality it's not far from it. Could it be time to push an alpha release :-)
Changed up how the Elexon reports are handled to make it simpler to add any that are needed. I need to work out how to handle the reports that return multiple blocks, e.g. DERBMDATA. Also need to look at how to handle the multi level results that get returned and how to easily access them in a sensible way. Not yet added any decene output to the command line app but hopefully will get to that soon.
I've added the start of a command line app and also a couple of the Elexon API commands. It's a work in progress but things should work and this seems to give me a base to build from in a similar manner to the python code I wrote. The app is basic and needs a lot of code sorted out and moved around :-)
I decided to make the entire form a private structure, so took the opportunity to flatten out the Form & FormData resulting in one single form. This works better and allows better integration between them. The updating of elements now works better and the form generation has been improved. DropDown elements can now have values set, but setting the Scheme to anything but the default results in a page redirect? Country seems to work and I will look at the others, but see no reason why setting the scheme should fail - but it does :-(
I'm still new to publishing go packages, so feel free to contact me with suggestions and improvements. :-)
All help welcome.