-
Notifications
You must be signed in to change notification settings - Fork 908
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to get paired name/size values for each snapshot associated with a VM? #2243
Comments
I think I found the correlation. I'll work on this over the next few days and will link to the results in case it is helpful to someone else. |
Scratch notes, since this seem as good a place as any to place them. Hierarchy of typesLevels above and some below skipped for simplicity and due to my ignorance.
Fields expanded
|
Unfortunately this didn't work. I ended up with the size of the vmsn files associated with the snapshots instead of the snapshots themselves. |
Downloaded, compiled and ran the I pulled this from page 155 of
This is starting to look like I'll need to match all of those file types in order to calculate snapshot size. The challenge will be tying the specific files back to a specific snapshot ID value. Digging further. I still can't escape the feeling that I'm overlooking something obvious. |
Looks like you're right. I tracked down the PowerCLI source and afaict it does what you describe above. You'll need to use DatastoreBrowser to get those file sizes. I think the only example we have of that is the govc |
Thanks for checking and confirming that.
I'll look there, thank you for the tip. |
I thought that maybe I could use the same approach I stubbed out earlier, but instead of limiting to snapshotData file type, I could use the types mentioned in the Unfortunately it doesn't appear that the I noticed that the snapshot VMDKs had a six digit suffix dot vmdk suffix (e.g., As the first file in the listing shows, knowing that pattern doesn't help that much. I'll go review the |
I took a look, but I don't understand how what I'm looking at maps to a file listing from a datastore. Once I get that, I'm not sure how I can reliably filter the list to only relevant files. Using the fields I listed previously, I believe I understand how
Any chance the source for the PowerCLI implementation can be shared? ;) I assume the answer is no, but could a pseudocode version be shared? |
Yeah, I can't share it, but looking at it again I realize the code using DatastoreBrowser is for older API versions.. there is a separate method for the modern versions of vSphere. Apologies for the confusion. It does still involve iterating over the LayoutEx structure as you've outlined above. I'll try to parse through it more and put something useful together when I get a chance. |
Thanks! |
Below is the converted C# to Go, hopefully this will be useful :) govmomi/object/virtual_machine.go Lines 647 to 648 in add8be5
Feel free to take the code and run with it: func extractDiskLayoutFiles(diskLayoutList []types.VirtualMachineFileLayoutExDiskLayout) []int {
var result []int
for _, diskLayout := range diskLayoutList {
for _, diskUnit := range diskLayout.Chain {
for i := range diskUnit.FileKey {
result = append(result, int(diskUnit.FileKey[i]))
}
}
}
return result
}
func removeKey(l *[]int, key int) {
for i, k := range *l {
if k == key {
*l = append((*l)[:i], (*l)[i+1:]...)
break
}
}
}
func SnapshotSize(info *types.VirtualMachineSnapshotInfo, parent *types.ManagedObjectReference, vmlayout *types.VirtualMachineFileLayoutEx, isCurrent bool) int {
var fileKeyList, parentFiles, allSnapshotFiles []int
diskFiles := extractDiskLayoutFiles(vmlayout.Disk)
for _, layout := range vmlayout.Snapshot {
diskLayout := extractDiskLayoutFiles(layout.Disk)
allSnapshotFiles = append(allSnapshotFiles, diskLayout...)
if layout.Key.Value == info.CurrentSnapshot.Value {
// The .vmsn file
fileKeyList = append(fileKeyList, int(layout.DataKey))
// The .vmdk files
fileKeyList = append(fileKeyList, diskLayout...)
} else if parent != nil && layout.Key.Value == parent.Value {
parentFiles = append(parentFiles, diskLayout...)
}
}
for _, parentFile := range parentFiles {
removeKey(&fileKeyList, parentFile)
}
for _, file := range allSnapshotFiles {
removeKey(&diskFiles, file)
}
fileKeyMap := make(map[int]types.VirtualMachineFileLayoutExFileInfo)
for _, file := range vmlayout.File {
fileKeyMap[int(file.Key)] = file
}
size := 0
for _, fileKey := range fileKeyList {
file := fileKeyMap[fileKey]
if parent != nil ||
(file.Type != string(types.VirtualMachineFileLayoutExFileTypeDiskDescriptor) &&
file.Type != string(types.VirtualMachineFileLayoutExFileTypeDiskExtent)) {
size += int(file.Size)
}
}
if isCurrent {
for _, diskFile := range diskFiles {
file := fileKeyMap[diskFile]
size += int(file.Size)
}
}
return size
} I tested with this change: modified govc/vm/snapshot/tree.go
@@ -25,6 +25,7 @@ import (
"github.com/vmware/govmomi/govc/cli"
"github.com/vmware/govmomi/govc/flags"
+ "github.com/vmware/govmomi/units"
"github.com/vmware/govmomi/vim25/mo"
"github.com/vmware/govmomi/vim25/types"
)
@@ -147,7 +148,7 @@ func (cmd *tree) Run(ctx context.Context, f *flag.FlagSet) error {
var o mo.VirtualMachine
- err = vm.Properties(ctx, vm.Reference(), []string{"snapshot"}, &o)
+ err = vm.Properties(ctx, vm.Reference(), []string{"snapshot", "layoutEx"}, &o)
if err != nil {
return err
}
@@ -161,6 +162,8 @@ func (cmd *tree) Run(ctx context.Context, f *flag.FlagSet) error {
}
cmd.info = o.Snapshot
+ size := SnapshotSize(o.Snapshot, nil, o.LayoutEx, true)
+ fmt.Printf("size=%s\n", units.ByteSize(size))
cmd.write(0, "", o.Snapshot.RootSnapshotList) |
@dougm Thank you! I appreciate the time/effort you spent to put this together. I'm going over the code now and comparing against my earlier efforts. |
No problem @atc0005 , it was straightforward to translate the code and follow the notes you'd outlined. Definitely not obvious how to calculate the snapshot size from the API doc, so it'll be great to have this as a reusable method within govmomi. |
Apply code provided by @dougm for testing purposes. Heavily annotated with questions/comments as I attempt to understand how the code works. This "understanding" is very much a work-in-progress. - credit: @dougm - refs vmware#2243
Thanks again for your help, and please let me know if my questions/commentary become tiresome. I'll switch over to other items and give you a break. :) func SnapshotSize(
info *types.VirtualMachineSnapshotInfo,
parent *types.ManagedObjectReference,
vmlayout *types.VirtualMachineFileLayoutEx,
isCurrent bool,
) int { I see that Later towards the end of the function the Here is a screenshot of the snapshots for a test VM in a vSphere 6.7 cluster: I've reverted to a snapshot named This probably makes sense in a literal way, but I would have expected the snapshot named Based on what I'm seeing here, if I want to start with a crawl of I'm still digging into this and hoping for the "ah ha!" moment. Once I have it, I suspect the direction will be obvious. EDIT: In case it is helpful in any way, I've taken the provided code and applied it against a branch in my fork (atc0005/govmomi@83bd29f). I've heavily annotated it as I attempt to figure out what each part of the code is doing. |
Hey @atc0005 , I don't mind helping, solving these types of mysteries can be fun. In any case, the func SnapshotSize(snapshot types.ManagedObjectReference, parent *types.ManagedObjectReference, vmlayout *types.VirtualMachineFileLayoutEx, isCurrent bool) int { And the diff below is how you would call it when walking the tree. Does PowerCLI have an option to display the size of each snapshot or just the total? modified govc/vm/snapshot/tree.go
@@ -25,6 +25,7 @@ import (
"github.com/vmware/govmomi/govc/cli"
"github.com/vmware/govmomi/govc/flags"
+ "github.com/vmware/govmomi/units"
"github.com/vmware/govmomi/vim25/mo"
"github.com/vmware/govmomi/vim25/types"
)
@@ -39,7 +40,8 @@ type tree struct {
fullPath bool
id bool
- info *types.VirtualMachineSnapshotInfo
+ info *types.VirtualMachineSnapshotInfo
+ layout *types.VirtualMachineFileLayoutEx
}
func init() {
@@ -78,7 +80,7 @@ func (cmd *tree) Process(ctx context.Context) error {
return nil
}
-func (cmd *tree) write(level int, parent string, st []types.VirtualMachineSnapshotTree) {
+func (cmd *tree) write(level int, parent string, pref *types.ManagedObjectReference, st []types.VirtualMachineSnapshotTree) {
for _, s := range st {
sname := s.Name
@@ -92,7 +94,10 @@ func (cmd *tree) write(level int, parent string, st []types.VirtualMachineSnapsh
names = append(names, sname)
}
+ isCurrent := false
+
if s.Snapshot == *cmd.info.CurrentSnapshot {
+ isCurrent = true
if cmd.current {
names = append(names, ".")
} else if cmd.currentName {
@@ -105,6 +110,12 @@ func (cmd *tree) write(level int, parent string, st []types.VirtualMachineSnapsh
var attr []string
var meta string
+ if true {
+ size := SnapshotSize(s.Snapshot, pref, cmd.layout, isCurrent)
+
+ attr = append(attr, units.ByteSize(size).String())
+ }
+
if cmd.id {
attr = append(attr, s.Snapshot.Value)
}
@@ -127,7 +138,7 @@ func (cmd *tree) write(level int, parent string, st []types.VirtualMachineSnapsh
}
}
- cmd.write(level+2, sname, s.ChildSnapshotList)
+ cmd.write(level+2, sname, &s.Snapshot, s.ChildSnapshotList)
}
}
@@ -147,7 +158,7 @@ func (cmd *tree) Run(ctx context.Context, f *flag.FlagSet) error {
var o mo.VirtualMachine
- err = vm.Properties(ctx, vm.Reference(), []string{"snapshot"}, &o)
+ err = vm.Properties(ctx, vm.Reference(), []string{"snapshot", "layoutEx"}, &o)
if err != nil {
return err
}
@@ -161,8 +172,9 @@ func (cmd *tree) Run(ctx context.Context, f *flag.FlagSet) error {
}
cmd.info = o.Snapshot
+ cmd.layout = o.LayoutEx
- cmd.write(0, "", o.Snapshot.RootSnapshotList)
+ cmd.write(0, "", nil, o.Snapshot.RootSnapshotList)
return nil
} |
I'm glad to hear that, I was concerned that I was encroaching too much on your time!
This is what I see in the UI for the top-most snapshot: Notice how it shows the disk usage as 20 GB? With the base disk size of 20 GB, this (incorrectly) led me to assume (the first time I saw it) that the base disk + snapshot = 40 GB. As I understand it, the potential is 40 GB if the snapshot completely fills out, but that's not what the UI suggests (to me anyway). PowerCLI seems to show the snapshot as size PowerCLI output snippet:
Full PowerCLI outputGet-VM RHEL7-TEST-RBD | Get-Snapshot | Select-Object -Property Name, Id, ParentSnapshotID, SizeMB, SizeGB | Format-List
Name : Fresh install, activation and patches
Id : VirtualMachineSnapshot-snapshot-18946
ParentSnapshotId :
SizeMB : 0.02670192718505859375
SizeGB : 0.0000260761007666587829589844
Name : 2019-10-15
Id : VirtualMachineSnapshot-snapshot-126800
ParentSnapshotId : VirtualMachineSnapshot-snapshot-18946
SizeMB : 8642.019138336181640625
SizeGB : 8.439471814781427383422851562
Name : 2019-12-05
Id : VirtualMachineSnapshot-snapshot-138143
ParentSnapshotId : VirtualMachineSnapshot-snapshot-126800
SizeMB : 6401.019138336181640625
SizeGB : 6.2509952522814273834228515625
Name : 2020-03-29
Id : VirtualMachineSnapshot-snapshot-163887
ParentSnapshotId : VirtualMachineSnapshot-snapshot-138143
SizeMB : 7953.019138336181640625
SizeGB : 7.7666202522814273834228515625
Name : Test Snapshot
Id : VirtualMachineSnapshot-snapshot-229096
ParentSnapshotId : VirtualMachineSnapshot-snapshot-18946
SizeMB : 1.018657684326171875
SizeGB : 0.0009947828948497772216796875
Name : Test Child snapshot
Id : VirtualMachineSnapshot-snapshot-229097
ParentSnapshotId : VirtualMachineSnapshot-snapshot-229096
SizeMB : 1.018657684326171875
SizeGB : 0.0009947828948497772216796875 Get-VM RHEL7-TEST-RBD | Get-Snapshot | Select-Object -Property Name, Id, ParentSnapshotID, SizeMB, SizeGB | Format-Table
Name Id ParentSnapshotId SizeMB SizeGB
---- -- ---------------- ------ ------
Fresh install, activation and patches VirtualMachineSnapshot-snapshot-18946 0.02670192718505859375 0.0000260761007666587829589844
2019-10-15 VirtualMachineSnapshot-snapshot-126800 VirtualMachineSnapshot-snapshot-18946 8642.019138336181640625 8.439471814781427383422851562
2019-12-05 VirtualMachineSnapshot-snapshot-138143 VirtualMachineSnapshot-snapshot-126800 6401.019138336181640625 6.2509952522814273834228515625
2020-03-29 VirtualMachineSnapshot-snapshot-163887 VirtualMachineSnapshot-snapshot-138143 7953.019138336181640625 7.7666202522814273834228515625
Test Snapshot VirtualMachineSnapshot-snapshot-229096 VirtualMachineSnapshot-snapshot-18946 1.018657684326171875 0.0009947828948497772216796875
Test Child snapshot VirtualMachineSnapshot-snapshot-229097 VirtualMachineSnapshot-snapshot-229096 1.018657684326171875 0.0009947828948497772216796875 In my use case I looped over all snapshots returned by To the question: the web UI suggests 20 GB as the value, PowerCLI shows I think it's based off of the snapshot data file size, which when I end up evaluating it with Go I am given 27999, or 27.3KB (after units.ByteSize() wrapped). Yesterday evening just before I pulled my hair out I finally got the stars to align and successfully (I think) calculated the size of multiple snapshot trees. I spent some more time this morning getting it together and I think I'll be ready to share the results in a day or so. Even so, it will still be in need of heavy refactoring. I think the net result is the same, though the process is handled a little differently. It's probably just a lack of understanding on my part. I'll apply the latest changes you provided tomorrow AM and study them. I appreciate your feedback and help with this. Quick question though: +if true { Did you mean to write |
Glad to hear you got the stars to align, look forward to your findings.
No, just meant that as a flag like the other attributes. So it should be something like |
Evidently I spoke too soon; looks like I have more testing to do. I had something working for the specific example that I've listed here, but my implementation fails to handle a single snapshot branch multiple levels deep where the active snapshot is the last in the chain. |
The more I troubleshoot this, the more my implementation is morphing to look like the provided logic. It seems that I didn't quite have a moment of insight like I thought I did. That said, it does seem like the current implementation matches the results as shown by the web UI, minus the first snapshot which is reported as the full disk size in the web UI, and just the snapshot data file per Go. Separate from govmomi, I am seeing odd results just between the size values for web UI and PowerCLI for the leaf snapshot (active). This is a different VM than previously shown in this issue. For this VM, the web UI shows 404.03 MB when selecting the name and 12.11 GB when selecting the "You are here" marker. The VM is powered on and is assigned 5 GB of RAM. PowerCLI shows:
It seems that PowerCLI is handling the calculations differently than the web UI's logic. I initially thought that the problem might be my PowerCLI version, so I upgraded it to the latest version. Get-Module | Where-Object {$_.Name -like 'VMware*'} | Select-Object -Property Name, Version
Name Version
---- -------
VMware.Vim 7.0.1.16997275
VMware.VimAutomation.Cis.Core 12.1.0.16997582
VMware.VimAutomation.Common 12.1.0.16997174
VMware.VimAutomation.Core 12.1.0.16997984
VMware.VimAutomation.Sdk 12.1.0.16997004 That's after a fresh removal, closure of open sessions, fresh install and another try. The results remained consistent between PowerCLI 11.x (previous) and 12.1.x (now). This almost seems like a case where the file keys associated with the VM's active state (growth since the last snapshot) are being ignored by the web UI, but recognized/reported by PowerCLI. Is this intentional? Is the differential disk growth between the last snapshot and the running state tracked in a manner visible to PowerCLI, but not the web UI / govmomi? When I run the calculations using Go/govmomi, I get the 404.03 MB value. I've not yet tested the latest provided changes with govc to see what it reports (that will be next). |
I dug further, and I found where I had a mistake in my logic. While I was getting the same results as shown by the web UI display, I wasn't getting the same results as the PowerCLI implementation. In particular, the PowerCLI implementation appears to combine the last (fixed) snapshot state with the current (growing) running state as one value. The web UI on the other hand displays a fixed value for the point-in-time that the snapshot was taken, then when you click into the Once I stepped through the initial code sample from #2243 (comment) again, I believe I now understand what is taking place. I probably should have just admitted that I wasn't following the logic in the provided code, but I felt like asking for doc comments to explain the logic would have been imposing on top of the time/effort that had already been expended to assist with this GH issue. Either way, I'm now getting consistent results across two different VMs with varying snapshot depths, so I'll work on cleaning the implementation up and then link to it. I still consider myself fairly new-ish to Go, so it's likely that I've overcomplicated things. It's also specific to my use case and not govc, but maybe it can still prove useful. |
CREDIT: This plugin would simply not have been possible without the help from @dougm. I'm grateful for both the general feedback (confirming I was looking in the right direction) and taking the time to craft code samples based on a review of the PowerCLI (C#) `Get-Snapshot` implementation. The logic used in this plugin is heavily inspired from the ideas presented, but *attempts* to use a slightly different (but probably less efficient) approach that made more sense to me. The current implementation is based on an evolution of my understanding of the API. While I believe the end results between the two implementations are the same, with further refactoring the implementation used by this project will probably end up looking nearly the same as the code examples originally presented. OVERVIEW: As with several other plugins in this project, this one borrows heavily from existing projects. In particular, this plugin was initially based on a PowerShell / PowerCLI plugin I wrote in 2019. In short, this plugin attempts to provide snapshot size details per snapshot for review, but evaluates size of snapshots for a VM based on the total of all snapshot size values, not individual values. This differs from the snapshots age plugin, which checks each snapshot individually to determine the service check result. Doc updates have been applied, example usage has been added, including a command definition "contrib" file illustrating how the plugin would be referenced within a production Nagios configuration. Partial work implemented with the snapshots age monitoring plugin to handle size monitoring has been completed and is functional as of this set of changes. Further refactoring and polish is needed, but based on initial use in our production environment the results appear to match the results provided by PowerCLI `Get-Snapshot` results. OTHER CHANGES: - Minor tweaks to snapshots age plugin to better mirror new snapshots size plugin. The idea is to eventually refactor both to share common code instead of replicating between the two. - Refactoring (more todo) of `internal/vsphere` code used by both plugins REFERENCES: - refs #4 - refs vmware/govmomi#2243 SEE ALSO: Note to self: See the following branches for "archival" commits that I hammered out while trying to understand the API. There is a lot of cruft and dead ends, but something there may be useful later. - `ARCHIVE-i4-add-snapshots-size-monitoring-plugin` - basically a "dirty" version of the branch holding this commit - `ARCHIVE-i4-add-snapshots-age-monitoring-plugin` - likely contains fragments of functionality from this branch before they were yanked to provide a more focused release (using what functionality was working at the time for the age-based checks) - `i4-add-snapshots-monitoring-plugin` - what I thought was going to be a combined plugin for age and size checks; abandoned, older state than the other branches Squash
CREDIT: This plugin would simply not have been possible without the help from @dougm. I'm grateful for both the general feedback (confirming I was looking in the right direction) and taking the time to craft code samples based on a review of the PowerCLI (C#) `Get-Snapshot` implementation. The logic used in this plugin is heavily inspired from the ideas presented, but *attempts* to use a slightly different (but probably less efficient) approach that made more sense to me. The current implementation is based on an evolution of my understanding of the API. While I believe the end results between the two implementations are the same, with further refactoring the implementation used by this project will probably end up looking nearly the same as the code examples originally presented. OVERVIEW: As with several other plugins in this project, this one borrows heavily from existing projects. In particular, this plugin was initially based on a PowerShell / PowerCLI plugin I wrote in 2019. In short, this plugin attempts to provide snapshot size details per snapshot for review, but evaluates size of snapshots for a VM based on the total of all snapshot size values, not individual values. This differs from the snapshots age plugin, which checks each snapshot individually to determine the service check result. Doc updates have been applied, example usage has been added, including a command definition "contrib" file illustrating how the plugin would be referenced within a production Nagios configuration. Partial work implemented with the snapshots age monitoring plugin to handle size monitoring has been completed and is functional as of this set of changes. Further refactoring and polish is needed, but based on initial use in our production environment the results appear to match the results provided by PowerCLI `Get-Snapshot` results. OTHER CHANGES: - Minor tweaks to snapshots age plugin to better mirror new snapshots size plugin. The idea is to eventually refactor both to share common code instead of replicating between the two. - Refactoring (more todo) of `internal/vsphere` code used by both plugins REFERENCES: - refs #4 - refs vmware/govmomi#2243 SEE ALSO: Note to self: See the following branches for "archival" commits that I hammered out while trying to understand the API. There is a lot of cruft and dead ends, but something there may be useful later. - `ARCHIVE-i4-add-snapshots-size-monitoring-plugin` - basically a "dirty" version of the branch holding this commit - `ARCHIVE-i4-add-snapshots-age-monitoring-plugin` - likely contains fragments of functionality from this branch before they were yanked to provide a more focused release (using what functionality was working at the time for the age-based checks) - `i4-add-snapshots-monitoring-plugin` - what I thought was going to be a combined plugin for age and size checks; abandoned, older state than the other branches
I was going to take the code you provided, apply it to the current Regarding code style, do you prefer to omit comments unless necessary? For example, convert code like this: // flat list of diskDescriptor, diskExtent values
var fileKeyList []int
// files associated with parent snapshot only
var parentFiles []int
// all snapshot files, regardless of whether snapshot has a parent
var allSnapshotFiles []int to this? var fileKeyList []int
var parentFiles []int
var allSnapshotFiles []int |
Apply further code changes provided by @dougm. - credit: @dougm - refs vmware#2243
Apply code provided by @dougm for testing purposes. Heavily annotated with questions/comments as I attempt to understand how the code works. This "understanding" is very much a work-in-progress. - credit: @dougm - refs vmware#2243
Apply further code changes provided by @dougm. - credit: @dougm - refs vmware#2243
Apply code provided by @dougm for testing purposes. Heavily annotated with questions/comments as I attempt to understand how the code works. This "understanding" is very much a work-in-progress. - credit: @dougm - refs vmware#2243
Apply further code changes provided by @dougm. - credit: @dougm - refs vmware#2243
Apply code provided by @dougm for testing purposes. Heavily annotated with questions/comments as I attempt to understand how the code works. This "understanding" is very much a work-in-progress. - credit: @dougm - refs vmware#2243
Apply further code changes provided by @dougm. - credit: @dougm - refs vmware#2243
- govc: Add size (`s`) flag to `snapshot.tree` subcommand - Add `object.SnapshotSize` and helper functions to support calculating snapshot size from other client code Credit to @dougm for providing successive code examples, explanation and guidance for understanding the required steps to implement this support. As noted on vmwareGH-2243, code provided by @dougm was (AFAIK) based heavily on existing C# code for PowerCLI implementation of the `Get-Snapshot` cmdlet. - credit: @dougm - refs vmware#2243
- govc: Add size (`s`) flag to `snapshot.tree` subcommand - Add `object.SnapshotSize` and helper functions to support calculating snapshot size from other client code Credit to @dougm for providing successive code examples, explanation and guidance for understanding the required steps to implement this support. As noted on vmwareGH-2243, code provided by @dougm was (AFAIK) based heavily on existing C# code for PowerCLI implementation of the `Get-Snapshot` cmdlet. - credit: @dougm - refs vmware#2243
Fix gosec G601 "Implicit memory aliasing in for loop" error by reassigning the iteration variable inside the loop. refs vmwareGH-2243 refs vmwareGH-2269
Fix gosec G601 "Implicit memory aliasing in for loop" error by reassigning the iteration variable inside the loop. refs vmwareGH-2243 refs vmwareGH-2269
Here is a PowerCLI snippet (either or):
which helped me generate a list of snapshot details like this from a Nagios plugin that I wrote about a year ago:
I'm looking to do the same thing with govmomi, but after digging the closest
I've found is #1197. I expect I'm overlooking something obvious.
Is the approach here to loop over
vm.LayoutEx.File
(wherevm
is amo.VirtualMachine
), filter type tosnapshotData
and build a mapping of sizes andkeys, then loop over
vm.Snapshot.RootSnapshotList
and pair together the idvalues here with key values from the other structure?
Thanks in advance for any pointers.
The text was updated successfully, but these errors were encountered: