Repository navigation
Arya Gujarathi's Notebook
Had virtual meeting with instructors. Covered syllabus, expectations for the class. Learned about what each subteam works with, and a brief overview of how electricity moved from power grids to individual homes.
Had a more detailed "lecture" on how a power substation works. Introduced myself to Sam Litchfield. Had intoductions with other students, and officially joined device modeling subteam. Had first outside of class meeting with subteam in which we split into smaller teams. I had my first introductions with Maria and Chungman. We discussed my role in presentation for next class.
Prepared and presented the introduction slide in Fall 20205 kickoff presentation. Listened to presentations of other subteam. Maria informed me of her conversation with Sam Litchfield about potential security issues in substations. Maria, Chungman, and I decided to use Google Docs instead of notion to take notes, as we are most familiar with Google Docs. We also each decided to research substations for the upcoming week.
Power Substations are filled with large and very expensive equipment. A power utilities’ greatest expense in substations is the power transformer. Other items and equipment involved in designing substations which are less expensive than power transformers are: power circuit breakers, circuit switchers, line-rupters, manual and motor controlled operated disconnects & bypass switches, potential / current transformers (PTs / CTs), lightning arresters, electric bus, metering equipment, steel structures, foundations, control switch house, main station service (SS), automatic transfer switch (ATS), protection & controls (P&C), relaying/IEDs, SCADA, telecommunications equipment, battery bank, control cables, fencing, grounding, lightning protection, real estate, and so on.
Maria and I had a conversation with Chungman about substations. He brought to our attention that various types of substations exist. The main 2 types are distribution and transmission. We had initially decided on attempting to model the distribution substation, however we thought Sam Litchfield might suggest we attempt to model a transmission substation, as we could better use it for simulation threats. We started to talk about components we would need to model our substation. After a long discussion with Sam, we decided upon some components we should include in our substation model. Specifically, PMUs and RTUs were the main devices discussed. And Sam suggested we continue with our initial idea of a distribution model for the power substation. The PMU is something important, but all of us (including Sam) understood it would be something hard to implement at first. With PMUs, there are many types and a variety of ways they could be implemented, so there are a variety of factors to consider. We needed to decide what deviecs we needed, and Sam gave us a clue with a diagram.
[Link](Power Substations are filled with large and very expensive equipment. A power utilities’ greatest expense in substations is the power transformer. Other items and equipment involved in designing substations which are less expensive than power transformers are: power circuit breakers, circuit switchers, line-rupters, manual and motor controlled operated disconnects & bypass switches, potential / current transformers (PTs / CTs), lightning arresters, electric bus, metering equipment, steel structures, foundations, control switch house, main station service (SS), automatic transfer switch (ATS), protection & controls (P&C), relaying/IEDs, SCADA, telecommunications equipment, battery bank, control cables, fencing, grounding, lightning protection, real estate, and so on.
https://3phaseassociates.com/basics-of-designing-power-substations/)
Traditional protocols (DNP3, MODBUS, proprietary)
Most of the early protocols in the electrical automation industry were proprietary protocols developed by device manufacturers. Although proprietary protocols work especially well with devices from the same manufacturer, lack of interoperability – along with vendor dependency – pushed electrical companies toward standard and open-source protocols. Today, device manufacturers have adopted popular standard protocols. As a result, proprietary protocols have been almost completely phased out from the industry. Like other protocols, substation automation communication protocols have evolved along with communication infrastructure improvement. Unlike slow and error-prone older protocols, newer protocols can deal with different communication mediums, recover from communication sever failures and deliver information in a more robust way.
Although older protocols such as MODBUS are still used in substation automation, most of the systems have already adopted protocols like DNP3 (North America) and IEC 60870 (Europe) as their de facto default protocol.
In addition to the set of rules, control headers and error recovery mechanism, traditional protocols also define the structure of a “point list.” A point list is the list of all data points the communication parties want to exchange as well as additional information such as point address and point type. The point list is defined during the configuration of a communication instance and will be deployed in the devices that will use that communication instance.
The most deployed protocol architecture in substation automation industry is the Master-Slave (server-client) architecture where one or several devices called slave (or server) are polled by a master (client) device or software in some predefined intervals. In some protocols, slaves can also initiate the communication to send information to the master using a mechanism called “unsolicited response.”
Although traditional protocols require more time and effort during configuration and commissioning, they are popular in the automation industry because they are easy to understand, configure and troubleshoot.
IEC 61850
Faster and more reliable network infrastructure opened the possibility of implementing higher-level protocols that make the task of configuration, commissioning and testing easier – even though the protocol itself is more complex. These newer protocols tend to move from an IT-oriented paradigm to an OT-oriented paradigm where users focus mostly on “what” a device should do rather than “how” it should do it.
In the early 1990s, parallel efforts were started to develop an object-oriented protocol that focuses more on the actual functions and information of a device, rather than low-level implementation detail such as register addresses and data type.
As utilities tried to move toward vendor-agnostic solutions, interoperability was another main force behind new protocol developments. New protocols should make sure devices from different vendors would be able to exchange information with the least amount of configuration.
The IEC 61850 standard was accepted by most of the utilities as a modern protocol that can address the shortcoming of traditional protocols. Unlike older protocols, IEC 61850 is more a suite of standards that address different aspects of a modern substation, rather than just a communication protocol. It defines in detail a standard model for each function in a substation plus the communication standards to support such a model as well the methods on how the map this model into the lower level communication. IEC 61850 also addresses necessary hardware requirements for a substation-grade device and defines a communication language that can be used to exchange a substation or a device model.
Although traditional protection systems tend to be completely separated from the automation and control system – and are still relying on dedicated hardwire signals between CTs and PTs and relays– IEC 61850 presents a system model where protection data points could be exchanged on a shared Ethernet link. It implements necessary measures to make sure this information will be delivered in a deterministic way within a redefined period of time.
GOOSE and sample values concepts in IEC 61850, define the object models and communication criteria that can be used to exchange protection information (e.g. voltage, current, breaker status) over a dedicated ethernet link called process bus (in less than 4 ms to comply with protection system time constraints). This reduces the amount of wiring in a protection system because all the wires between CTs, PTs and protection relays can now be merged into one ethernet cable.
IEC 61850 also includes testing methods a user can refer to during the commissioning or maintenance phase of a project to make sure all devices are functioning according to the requirement of the project – and to isolate problems during a troubleshooting session.
Physical communication: Modem, serial, copper ethernet , fiber, radio, cellular Being connected to a substation and retrieving vital information from remote devices has always been a challenge for system designers. Not all the substations are of the same size or importance. Many substations are in remote areas where communication could be the biggest challenge when it comes to monitoring a substation.
Early remote monitoring started by using modems on telephones or leased lines. At the time when most of the devices in the field had very limited communication capabilities, these methods of communication were sufficient for most situations. Initial efforts were made using a gateway device in the field that would concentrate information it was receiving from serial devices and send that information to a master station using a modem based on a predefined time schedule. Concentrating information would improve the communication (and reduce the cost) as sending data points in one burst would reduce the communication time – as compared to sending small data packets from different devices over a longer period of time.
In modern substations, most devices communicate via ethernet links. Data from different devices is sent to the control centers via various communication mediums. Utilities generally prefer to install their own inter-sub communication infrastructure using fiber-optic links or radio mesh systems but in some cases, especially in remote areas or smaller substations, using cellular modems becomes more practical.
Although using a public infrastructure such as cellular networks can reduce maintenance costs, it also raises concerns about security and availability.
Software of Substation article
To do:
- Decide where to start with modeling substation
- Research devices that should be used in our initial model, and purpose of each
We have to find a "bare bones" substation model. We agreed to start from the very basics.
Once we have a framework that works well and simple, then we can start to add more complicated devices and make it more functional to model real-world threats. We agreed to continue researching various aspects of modeling substations, to broaden our understanding of the field before we dig in.
https://pages.mtu.edu/~avsergue/EET3390/Lectures/CHAPTER6.pdf
https://www.sciencedirect.com/topics/computer-science/distribution-substation
From researching and reading a few articles, I was able to get a basic understanding of what needs to be put in our bare bones substation. I learned that sensors are crucial components in substations since they measure electrical parameters that describe the live, real-time state of the substation. They monitor the state of equipment and send accurate data to IEDs, which as Intelligent Electronic Devices. Switches, asa simple as they sound, are also crucial. They open or close circuits, which allow systems to control the flow of electricity. Usually, these switches are opened or closed based on data from sensors, which was the previous device mentioned. Sensors collect data on a variety of things. They can reflect changes in temperature, voltage, and even the minute vibration of equipment!
It can be said that sensors are very important, as the values they read control many other important parts of the substation. This is why sensors definitely belong in a basic model of our substation, and we will definitely attempt to include it.
To do:
- Midterm Paper
- More research on devices needed
- More research on RTU specifically
- Diagram models and Networking brain dump
To be completely honest, most of the time I dedicated this week towards VIP was for the Midterm Paper and Presentation. As a first semester student of the VIP, I made sure to review the Final paper of last semester. I wanted to summarize what I read in last semester's final paper, specifically what related to power substations. I read about a 4-wheel-drive robot that can traverse on uneven terrain. This robot is an inspection robot for power substations. It can have its own sensors to detect valuable information that the power substation sensors alone cannot detect. The robot also uses LiDAR for navigation Thermal camera is also used for night vision.
I then made some additions to the midterm paper, after speaking to Maria. Around half of the content on this page was updated by me:
I had never worked on a group midterm paper before. This was good practice for me, especially getting to use the overleaf software for the first time. I also prepared some of my part for the midterm presentation.
We were still unsure about where to start with our "bare bones" substation, so Sam guided us with a rough sketch we drew.
During our meeting, Maria, Chungman, and I discussed another diagram we found online.
Maria and Chungman discussed some networking things such as DNP3, which I realized I needed to ask more question about to get a better understanding of. I learned it was a networking protocol used to connect and control devices. It allows for efficient transfer of data between master substations.
To do:
- Midterm Presentation
- Continue to develop substation model diagram
Continue to develop slides and prepare for Midterm Presentation
Presented the midterm presentation. I feel our team did a nice job, we were well prepared and confident of our findings that we presented. I was able to discuss our decision use chose the distribution model of substation, and our plans to develop a bare bones model and eventually add more devices. Since I spent most of the week preparing for this, I wasn't able to focus as much as I would liked to on the other To dos. So more on those next week.
During our meeting, Maria and I collectively decided we need to research which devices talk to the master in a substation. The master serves as a control center in a power substation. It can turn on and off certain devices, and receive data from them, especially sensors. We needed to figure out some basic devices to add to our model so that they can interact with the master.
To do:
- Continue research on devices needed for model
- Continue to develop substation model diagram
- Look into which devices are needed (extra focus on RTU) https://library.e.abb.com/public/7ce525aadd2a42588ca676cf7d125a61/3BUS095131_en_IEC61850_Overview_and_Benefits_Paper_General.pdf
We agreed to continue research on devices needed for model.
Me and Maria spoke about what devices our research would entail. We came to the conclusion that we should probably focus extra on the RTU.
Based on the articles and research papers I read, I found (assuming the substation uses legacy SCADA) that the primary device the master retrieves information from is the RTU. The RTU (Remote Terminal Unit) serves as a important device, since it gathers information from equipment like transformers, circuit breakers, and sensors. It converts signals from analog to digital form before it sends those results to the master control system. The RTU also allows for remove controlling of substation functions. It can send status updates and quick measurements to uphold strict response time requirements. Lastly, RTUs can receive control signals from the master, which can include instructions to open or close circuit breakers. Overall, RTUs help automate a lot of the operations of modern substations. This is the crucial device that communicates with the master of a power substation, even in the distribution model.
PMUs measure voltage, frequency, and rate of change of frequency. Every single measurement is time-stamped with a value synchronized by the GPS. This allows for phasor data from many substation to all be aligned. Communication standards such as IEEE C37.118 exist to transfer synchronized data. Their relation to the master will be discussed shortly.
PDCs (Phasor Data Concentrators) serve to collect data from many PMUs simultaneously. PDCs time align and filter Phasor data from multiple PMUs, each streaming its own high-rate data.
The master of the substation typically doesn't get data from PMUs directly. Rather, the PDC acts as an intermediate data gatherer. Data is streamlined and aggregated from many PMUs to a single PDC (using IEEE C37.118), from which the PDC forwards this filtered data to the master (sometimes using C37.118).
Gateways (aka protocol converters) reside in between field devices and the master. They translate protocols and aggregate data, so that the master doesn't have to worry about handling multiple types of protocols from different devices for data transfer. From the master's perspective, the gateway looks like a single RTU or IED. To the field devices, the gateway looks like another master. The master can effectively use a single protocol to communicate with with many other devices, even though if they have other protocols, due to the addition of the gateway. The gateway serves as a translator.
To do:
- Look at code for existing devices, to understand how they already work
- Attempt to create code for RTU devices
We decided that the main device we needed to connect to the master was the RTU. This was consistent across various models of distribution substations; the RTU was essential.
After discussing with Sam, we understood that the bare bones parts of our transformer would be a few circuit breakers, a transformer, and a disconnector switch.
We pondered the idea of just updating the master to include RTU functionality into its current functionality, however Sam suggested we don't do that. Sam told us there are other devices that connect to the RTU, and he drew this on the white board and wanted us to figure out the implementation part ourselves.
Maria sent me the GitHub link for the code of our existing devices. I wanted t dedicate time to understanding the existing ones before I attempted to develop new ones.
I began to review the basic structure for all devices. I soon realized that all devices should have certain functions in common. Specifically, initialize, activate, deactivate, and destroy, and getName. Only some devices implemented such methods though. For example, while initializing the switchGear device, one must set the name, protocol, various voltage ratings, status, and whether the gear is tripped.
Initializing the bay controller on the other hand, required various voltage value to be stored, and even a voltage history. Each controller also needed to store multiple bay devices.
The master was different from of the devices, as that it specialized in received certain types of inputs and handlers. It dealt with communication way more than the other devices. It would also handle various channels.
In order to start writing code for the RTU device, I still felt I needed to better understand our existing devices. I would like to speak with Maria and Chungman more to figure this part out, I can't figure it out myself.
To do:
- Install and set up nix development environment
- Update changes and develop nix
- Build HELICS and GridLAB-D and test out examples (partially complete with errors)
Kaden gave us some installation instructions he wanted us to test out.
I follow our VIP's GitHub instructions for a clean install of nix and the required dependencies on Macs. It took longer than expected, and there were some leftover traces of old nix files that I had to delete before I could get this to work. Debugging took some time. I finally got the nix setup working:
Following the instructions from Kaden's Notebook, I then attempted to clone the repo again. I tried that for a while and ran into many issues down the line, so I then decided to nuke the existing cps-cosimulation-env repo. I then recloned, and went to the networking branch. I had some issues with updating nix flake, but I was able to get that to work. The "make all-install" suceeded, however the examples still didn't work. When I ran the simple_gridlab_example, it hung, with just these 4 lines.
I killed it, and some log files were produced when I checked the results directory. However, these files were empty.
DNP3 had its own set of issues, being unable to run. It kept on saying uv command not found, even though I was using the latest version of python and on the latest repo branch, with Sam and Kaden suggested as potential fixes.
Sam later came and tried running some other commands in my terminal. We stumbled upon a segfault, which Sam told me was known for GridLabD. He told me this has been a known issue on Mac for a few years, and they even released a "fix". which our VIP pulled into our code, but it still didn't work well. There was no way to continue proceeded on the new Macs with nix, Sam said. We should look into other ways for the future.
- Attempt to get GridLab-D running on Mac using Rosetta to emulate x86_64 on Apple Silicon
Kaden doesn't have a Mac, but since Chungman and I both have Macs, he told us to attempt this. Kaden explained to us that this is the solution the GridLab-D people gave themselves, and we should attempt to replicate by following these instructions. He gave us another article which told us to set up nix, as I figured there might have been some initial issues with how I installed nix.
I followed all the steps to install nix, and with a little trouble I as able to get nix successfully installed. The initial page Kaden provided was many people commenting they had similar Grid-Lab D issues on Mac, and that there is some issue that "cannot currently be fixed without a significant update to GridLAB's inter-module communication architecture." One fix that was provided was to use Rosetta, which allows a Mac with Apple silicon to use apps that were made for an Intel Processor Mac. The Rosetta installation and execution didn't go too well for me. I made multiple attempt, spending a short amount of time over a few days to no avail. I wish I could have spent longer debugging but I had multiple exams this week. Chungman also ran into similar setup issues, here are some screenshots he sent me.
More unfruitful attempts at debugging
We were running out of options at this point. Kaden had one last idea. Kaden told us that the "Networking_team branch has been updated to move from Nix to Docker." He sent us the README at this link, and told us to delete our existing clone of the repo before attempting the README steps. He said a git pull might work, but it would be safer to reclone, which he had tested.
I was able to get through all the installing steps in the README file, without any major issues. Post installation steps went very smoothly for me too. My issues started when I tried running the GridLab-D example. This is the error I got and I wasn't able to debug it. Chungman and Kaden weren't sure how to handle it either.
At least 3 of us attempted to run the Grid-Lab D example. Me, Chungman, and 1 or 2 others. All of us made it through the installation and post-installation steps, but running the Grid-Lab D example is what gave us trouble. Now the surprising part was that we all were facing different issues even though we were all on MacOS. My issue was quite different than Chungman's, as was his quite differen't than mine. We both have different chips, as I have M3 and I believe Chungman has M1. Maria was on a Window's laptop, and the example wasn't working properly for her either!
Sam understood our struggles and frustration very well. He told us we shouldn't have to spend so much time debugging, we should be able to actually test out examples and write new code to test, which we are currently unable to do. A lot of this delayed our work on substation model development, which me, Maria, and Chungman wanted to spend more time on. Sam gracefully assured us that next semester we will have multiple laptops available for use during VIP session, all with the exact same hardware and OS, so that any debugging or fixes done for one computer will be done for all, and changes across laptops will be more smooth. We can spend less time debugging and more time actually debugging.
- Final Presentation
- Peer Evals
We discussed ideas about what to include in our slides for the final presentation.
I completed my Peer Evals, and I completed my slides.
I also presented my slides during the meeting for the final presentation. In the first slide, blue represents goals we accomplished this semester, and red represents goals we have yet to accomplish, either short term or long term.
The second slide is stuff I put together based on the fact that I am one of few returning VIP members on my team, most of the others are graduating or finished their 3 VIP semesters. It was some feedback and suggestions I had to make work flow more smoothly next semester, a lot of which was already mentioned by Sam.
Thanksgiving Break
Work on Final Paper. All additions by me (both text and bib) can be seen here.
Wanted to thank Sam and William for such a great semester. You were both always more than happy to assist me.