Fix Unix FileStatus flag retrieval without perf regression #47348

carlossanlop · 2021-01-22T19:55:16Z

If you are reviewing this PR, please take a look at each commit individually (small incremental changes).

FileSystemEntry and FileSystemInfo were not reporting their hidden flags correctly on Linux and MacOS.

On Linux and MacOS, the hidden attribute can be set by prefixing the file or folder name with a dot.

On MacOS and Windows, the hidden attribute can be set by prefixing the Hidden flag on the file or folder.

The original PR introduced a perf regression due to a new stat call.

The FileStatus and FileSystemEntry initialization code had duplicate code when setting flags. The right way to fix it is to move all the stat and lstat calling inside FileStatus.

FileSystemEntry also has some special needs when detecting directories and symbolic links: since it has access to the InodeType, some info can be inferred from them, saving stat calls. So that logic had to be preserved in the FileSystemEntry initialization method, while deferring as much as possible to FileStatus.

Across FileStatus, I ensured the stat and lstat calls only get executed lazily (don't call again if already initialized), unless explicitly asking to refresh them (forcing the new stat call).

I also added the original unit tests to verify both the hidden flag and the readonly flag (we didn't have unit tests for the last one).

I made sure all unit tests are passing in MacOS I also ran a temporary unit test with the below code to quickly verify that the performance regression is gone:

Temporary perf test

[Fact]
public void Test()
{
  var sw = new Diagnostics.Stopwatch();
  // Inside the `experiments` folder, I created one symbolic link that points to a folder,
  // and one symbolic link that points to a file
  // This ensures the `IsSymbolicLink` method I added is called.
  string path = "/Users/calope/Public/repos/experiments";
  string pattern = "*";

  Console.WriteLine("Starting...");
  sw.Start();
  for (int i = 0; i < 7000; i++)
  {
     // This is the exact same code we have in a FileSystem benchmark:
     // https://github.com/dotnet/performance/blob/a25a963e3f14214102207a2b75a5ece25162a333/src/benchmarks/micro/libraries/System.IO.FileSystem/Perf.Directory.cs#L117
      Directory.EnumerateFiles(path, pattern, SearchOption.AllDirectories).Count();
  }
  sw.Stop();
  Console.WriteLine($"Elapsed: {sw.ElapsedMilliseconds}");
}

cc @iSazonov

Pending: I want to get official dotnet/performance results for all the IO.FIleSystem performance tests. But I need to figure out how to run the micro benchmarks in .NET 6.0 in MacOS, because the Preventing regressions instructions are not working anymore. I get this error:

Error message

calope@Carloss-MacBook-Pro performance % cd src/benchmarks/micro 
calope@Carloss-MacBook-Pro micro % dotnet run -c release -f netcoreapp5.0 --artifacts "~/Public/perf_after" --corerun "~/Public/repos/runtime/artifacts/bin/testhost/net6.0-OSX-Release-x64/shared/Microsoft.NETCore.App/6.0.0/corerun" --filter System.IO.Tests.Perf_Directory.EnumerateFiles
/usr/local/share/dotnet/sdk/5.0.102/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.TargetFrameworkInference.targets(141,5): 
error NETSDK1045: The current .NET SDK does not support targeting .NET Core 6.0.  
Either target .NET Core 5.0 or lower, or use a version of the .NET SDK that supports .NET Core 6.0. 
[/Users/calope/Public/repos/performance/src/benchmarks/micro/MicroBenchmarks.csproj]
The build failed. Fix the build errors and run again.

Edits:

I had originally reported a perf improvement with my original changes. Unfortunately, they also had a bug uncovered by the CI, which I'm investigating.
CI issues fixed. Duration of the test before and after my changes:
- Without changes: 44894
- With changes: 46904

…zedMainCache

…' cases, but not failure. Use _secondaryCache and _initializedSecondaryCache.

…yExistIndependentlyOfTarget unit test (link should exist after main file deleted).

Need to preserve the _isDirectory field. The order and the place in which it acquires a value matter. It should also be set to false when EnsureStatInitialized is called. The unit test that verifies this is System.IO.Tests.DirectoryInfo_Exists.SymLinksMayExistIndependentlyOfTarget.

… both usage cases.

…stance.

…nstance.

….ToFileSystemInfo, since it's the only place where it is used. Move the Debug.Assert to the top with the ROS<char> instead of the string.

stephentoub · 2021-01-22T20:02:09Z

If you are reviewing this PR, please take a look at each commit individually (small incremental changes).

Which of the 28 commits here fixes the actual issue? Lots of these looks like style.

carlossanlop · 2021-01-22T20:28:39Z

Which of the 28 commits here fixes the actual issue? Lots of these looks like style.

Only a few were style. This is the commit that fixes the hidden bug: 62b77b8

carlossanlop · 2021-01-22T20:45:35Z

The CI is failing in OSX in tests related to the hidden flag. I'm investigating.

Strangely, they are passing in my MacOS.

Edit: I'm making it a draft while I address the failures.

…eSystemEntry.Initialize calls to refresh individual caches.

… hidden check in Initialize to prevent perf regression.

src/libraries/System.IO.FileSystem/tests/Enumeration/AttributeTests.cs

adamsitnik

The right way to fix it is to move all the stat and lstat calling inside FileStatus

This is a very good idea. 👍

The changes look good, I would just improve the naming a little bit and encapsulate the caching logic within the FileStatus type.

As soon as you fix the CI errors I am going to perform a more detailed review.

I get this error

As mentioned offline, you get an error about missing .NET 6 SDK. Perf repo always requires latest SDK, so we can have benchmarks that test new APIs (https://github.com/dotnet/performance/blob/master/docs/prerequisites.md#net-core-sdk)

adamsitnik · 2021-01-25T09:58:29Z

src/libraries/System.IO.FileSystem/tests/Enumeration/SkipAttributeTests.cs

+            if (useHiddenFlag)
+            {
+                fileTwo.Attributes |= FileAttributes.Hidden;
+                fileFour.Attributes |= FileAttributes.Hidden;
+            }


just for my education: is it possible to set these attributes before creating the file and have FileInfo.Create create the file with the attributes provided?

No. The file needs to exist. The Attributes setter calls the FileStatus.SetAttributes method (you can see it in this PR), which sets each flag into the file via a P/Invoke.

adamsitnik · 2021-01-25T10:00:55Z

src/libraries/System.IO.FileSystem/tests/Enumeration/SkipAttributeTests.cs

+
+        [Fact]
+        [PlatformSpecific(TestPlatforms.AnyUnix)]
+        public void SkippingHiddenFiles_Unix()


nit: I think that the test name should include the information about using prefixes.

Suggested change

public void SkippingHiddenFiles_Unix()

public void SkippingFilesHiddenWithDotPrefixOnUnix()

adamsitnik · 2021-01-25T10:02:08Z

src/libraries/System.IO.FileSystem/tests/Enumeration/SkipAttributeTests.cs

@@ -21,25 +21,42 @@ protected virtual string[] GetPaths(string directory, EnumerationOptions options
        }

        [Fact]
-        public void SkippingHiddenFiles()
+        [PlatformSpecific(TestPlatforms.Windows | TestPlatforms.OSX)]
+        public void SkippingHiddenFiles_Windows_OSX()


nit: I think that the test name should include the information about using attributes.

Suggested change

public void SkippingHiddenFiles_Windows_OSX()

public void SkippingFilesHiddenWithAttributesOnWindowsAndOSX()

What happens when the user sets the attribute on Linux? Is it just ignored and the user gets silent error?

No error. It just gets ignored. I just tested it on Ubuntu.

No error. It just gets ignored.

I wonder if we should address that somehow. Usually, we would just mark the enum member with UnsupportedOSPlatform

public enum FileAttributes { [UnsupportedOSPlatform("Linux")] Hidden }

But if I understand correctly, the enum can be returned by File.GetAttributes for files that start with a dot on Linux, but has no effect when used with the corresponding setter. Should we at least mention it in the docs?

adamsitnik · 2021-01-25T10:05:10Z

src/libraries/System.IO.FileSystem/tests/Enumeration/AttributeTests.cs

+            IEnumerable<string> enumerable = new FileSystemEnumerable<string>(
+                 testDirectory.FullName,
+                 (ref FileSystemEntry entry) => entry.ToFullPath(),
+                 new EnumerationOptions() { AttributesToSkip = 0 })


does setting AttributesToSkip to 0 here has any effect on the test behavior? Would removing it change anything?

Suggested change

new EnumerationOptions() { AttributesToSkip = 0 })

new EnumerationOptions())

Yes, the IsHiddenAttribute method starts failing if I don't explicitly set this to 0 (to signify "don't skip any attributes").

The reason is because when unset, then by default we skip the Hidden and the System attributes. That's how EnumerationOptions.AttributesToSkip is documented.

Skip entries with the given attributes. Default is FileAttributes.Hidden | FileAttributes.System.

adamsitnik · 2021-01-25T11:00:19Z

src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEntry.Unix.cs

+            entry._status = default;
+            entry._status.Invalidate();


would it be possible to encapsulate the logic that is modifying the FileStatus below and move it into FileStatus itself? So the FileStatus could keep all the caching logic as it's private detail and return only a valid FileStatus?

Suggested change

entry._status = default;

entry._status.Invalidate();

entry._status = FileStatus.Create(entry, directoryEntry, out FileAttributes attributes);

adamsitnik · 2021-01-25T11:18:30Z

src/libraries/System.IO.FileSystem/src/System/IO/FileStatus.Unix.cs

-            else if (_fileStatus.Gid == Interop.Sys.GetEGid())
+        }
+
+        internal bool HasSecondaryDirectoryFlag => HasDirectoryFlag(_secondaryCache);


HasSecondaryDirectoryFlag does not tell me much. Is it checking if the symlink is a directory?

Suggested change

internal bool HasSecondaryDirectoryFlag => HasDirectoryFlag(_secondaryCache);

internal bool IsSymlinkToDirectory => HasDirectoryFlag(_secondaryCache);

Yes, it is checking if the symbolic link's target is a directory.
Edit: I renamed it appropriately, based on the other related feedback.

adamsitnik · 2021-01-25T11:19:51Z

src/libraries/System.IO.FileSystem/src/System/IO/FileStatus.Unix.cs

+
+        private bool IsValid =>
+            _initializedMainCache == 0 && // Should always be successfully refreshed
+            (_initializedSecondaryCache == -1 || _initializedSecondaryCache == 0); // Only refreshed when path is detected to be a symbolic link


Suggested change

(_initializedSecondaryCache == -1 || _initializedSecondaryCache == 0); // Only refreshed when path is detected to be a symbolic link

_initializedSecondaryCache <= 0; // Only refreshed when path is detected to be a symbolic link

0 means successful retrieval. -1 means uninitialized. Any other value means it's a linux error number. Although I don't think error numbers can be negative, I wanted to be very specific about the two possible values I cared about.
I assume it has a slight performance improvement to do one check instead of two?

I assume it has a slight performance improvement to do one check instead of two?

If it has any, it's most probably very, very small. I was more thinking about simplifying the code. As long as everything is encapsulated within this type, we should rather not worry about values that are negative, but not equal -1 (as we control everything in this type and it's impossible). But it's a minor nit.

adamsitnik · 2021-01-25T11:22:19Z

src/libraries/System.IO.FileSystem/src/System/IO/FileStatus.Unix.cs

+
+        internal bool IsSecondaryCacheValid => _initializedSecondaryCache == 0;
+
+        private bool IsValid =>


nit: When I see FileStatus.IsValid I think of the file being valid, not file status cache. Would HasValidCache be a better name?

Suggested change

private bool IsValid =>

private bool HasValidCache =>

Sure. I'll use HasValidCaches (plural).

src/libraries/System.IO.FileSystem/src/System/IO/FileStatus.Unix.cs

adamsitnik · 2021-01-25T11:28:44Z

src/libraries/System.IO.FileSystem/src/System/IO/FileStatus.Unix.cs

        }

-        internal void SetLastWriteTime(string path, DateTimeOffset time) => SetAccessOrWriteTime(path, time, isAccessTime: false);
+        internal void RefreshCaches(ReadOnlySpan<char> path)


as mentioned before, it would be great if we could move all the methods that need to call this particular method inside FileStatus and make it private, so the cache would become an implementation detail to the consumers

There is a difference between called EnsureCachesInitialized and RefreshCaches: the former will check if they are initialized, and if not, it will refresh them. The second one will always refresh them.
Maybe I can pass a boolean to indicate if we want to force refresh or not. That way, only one method is visible outside.

iSazonov · 2021-01-25T11:26:25Z

src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEntry.Unix.cs


        public FileSystemInfo ToFileSystemInfo()
        {
+            Debug.Assert(!PathInternal.IsPartiallyQualified(FullPath), "FullPath should be fully qualified when constructed from directory enumeration");


Debug.Assert in public API? If it is important check it should be a real time check.

Related to the other comment below, I decided to revert this change since it was unrelated to the purpose of this PR.

iSazonov · 2021-01-25T11:27:15Z

src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEntry.Unix.cs

+            FileSystemInfo info = IsDirectory
+                ? new DirectoryInfo(fullPath, fileName: fileName, isNormalized: true)
+                : new FileInfo(fullPath, fileName: fileName, isNormalized: true);
+
+            info.Init(ref _status);


Why not create internal ctor-s with the parameter?

I guess I can revert this change. Originally it was not a constructor but a factory method. I thought we could save one method call if I put all that logic in the only place where it was called. But it's unrelated to this change anyway.

iSazonov · 2021-01-25T11:41:56Z

src/libraries/System.IO.FileSystem/src/System/IO/FileStatus.Unix.cs

+        }
+        private void ThrowOnCacheInitializationError(ReadOnlySpan<char> path)


Suggested change

}

private void ThrowOnCacheInitializationError(ReadOnlySpan<char> path)

}

private void ThrowOnCacheInitializationError(ReadOnlySpan<char> path)

iSazonov · 2021-01-26T03:40:01Z

src/libraries/System.IO.FileSystem/src/System/IO/FileStatus.Unix.cs

+
+        // We track intent of creation to know whether or not we want to (1) create a
+        // DirectoryInfo around this status struct or (2) actually are part of a DirectoryInfo.
+        internal bool InitiallyDirectory { get; set; }


It is seem assigned only in ctor.

Suggested change

internal bool InitiallyDirectory { get; set; }

internal bool InitiallyDirectory { get; private set; }

It is seem assigned only in ctor.

So it should be { get; } instead of { get; private set; } ?

iSazonov · 2021-01-26T03:40:52Z

src/libraries/System.IO.FileSystem/src/System/IO/FileStatus.Unix.cs

            // IMPORTANT: Attribute logic must match the logic in FileSystemEntry

-            EnsureStatInitialized(path);
+            EnsureCachesInitialized(path);


Do we need the comment? Is all logic in one place now?

iSazonov · 2021-01-26T03:42:52Z

src/libraries/System.IO.FileSystem/src/System/IO/FileStatus.Unix.cs

-                        Interop.CheckIo(Interop.Sys.LChflags(path, (_fileStatus.UserFlags & ~(uint)Interop.Sys.UserFlags.UF_HIDDEN)), path, InitiallyDirectory);
-                    }
-                }
+                return DateTimeOffset.FromFileTime(0);


Can we return const?

iSazonov · 2021-01-26T03:45:40Z

src/libraries/System.IO.FileSystem/src/System/IO/FileStatus.Unix.cs

+            if (!IsMainCacheValid)
+            {
+                if (!TryRefreshMainCache(path))
+                {
+                    if (!continueOnError)
+                    {
+                        ThrowOnCacheInitializationError(path);
+                    }
+                }
+            }


Can we use one if? Or move all the logic in TryRefreshMainCache()?

I can move it all to one if. I don't think we should throw from within a try method.

I mean IsMainCacheValid check could be in TryRefreshMainCache().

iSazonov · 2021-01-26T03:49:33Z

src/libraries/System.IO.FileSystem/src/System/IO/FileStatus.Unix.cs

        private unsafe void SetAccessOrWriteTime(string path, DateTimeOffset time, bool isAccessTime)
        {
            // force a refresh so that we have an up-to-date times for values not being overwritten
-            _fileStatusInitialized = -1;
-            EnsureStatInitialized(path);
+            Invalidate();


Invalidate that? Can we use more informative name?

carlossanlop · 2021-01-29T17:41:54Z

Had a conversation with @adamsitnik about this. I am going to look into splitting the Unix code into Linux and OSX, because it may be possible to use an API (at least on Linux) that lets us retrieve the list of files inside a folder with the hidden flag specified. This will allow us to have separate performance results for those platforms. I will do the same investigation for the Windows API.

iSazonov · 2021-02-11T11:44:55Z

This could fix #31301 too.

iSazonov · 2021-02-25T05:59:23Z

Currently I have worked on improving PowerShell to handle OneDrive. It is a reparse points. And we need to analyze a surrogates. This information is not present in FileSystemInfo.FileAttributes - there is only one flag RepasePoint but we need more and we have to call FindFirstFileEx and check dwReserved0.

Thus, we need extended attributes not only on Unix, but also on Windows. Perhaps we could introduce new FileExtendedAttributes property.
This would not only allow us advanced scenarios but also help solve the performance problem.

ghost · 2021-03-31T10:00:19Z

Draft Pull Request was automatically closed for inactivity. It can be manually reopened in the next 30 days if the work resumes.

carlossanlop added 28 commits January 21, 2021 00:01

New hidden and readonly tests

a50c0fc

FileStatus.Unix.cs public->internal

b8f69b7

FileStatus.Unix.cs sort members

84433dd

FileStatus.Unix.cs brackets

5070cde

Add VerifyStatCall method

0686368

Use VerifyStatCall with existing FileStatus Lstat call

5e5932b

FileStatus.Unix.cs sort Initialize method

a866e73

Rename _fileStatus => _mainCache, _fileStatusInitialized => _initiali…

48d0d5a

…zedMainCache

Add _secondaryCache and _initializedSecondaryCache

96dd50c

Add IsValid. Use it where value of _initializedMainCache is checked.

f6b35c3

Update EnsureStatInitialized to check both _initialized* values

f865030

Adjust VerifyStatCall to consider ENOENT, ENOTDIR as 'path not exists…

334b561

…' cases, but not failure. Use _secondaryCache and _initializedSecondaryCache.

Add and consume IsSymbolicLink and HasSymbolicLinkFlag.

e8e5d9b

Fix Refresh stat call error verification to prevent bug in SymLinksMa…

82c382f

…yExistIndependentlyOfTarget unit test (link should exist after main file deleted).

Use Invalidate to reset _initialized* values

4dbb740

Add HasReadOnlyFlag and consume in GetAttributes and IsReadOnly

b3cb947

Add HasHiddenFlag and IsHidden and consume in SetAttributes

a69901b

Remove Initialize. Call Invalidate and InitiallyDirectory directly on…

6ba7c79

… both usage cases.

FileSystemEntry.Initialize should get stat info via its FileStatus in…

4b3aec0

…stance.

FileSystemEntry.Initialize should get lstat info via its FileStatus i…

75552fe

…nstance.

Combine FileSystemEntry.IsHidden dot check with flag check.

62b77b8

Move internal static FileSystemInfo.Create logic into FileSystemEntry…

ac7062b

….ToFileSystemInfo, since it's the only place where it is used. Move the Debug.Assert to the top with the ROS<char> instead of the string.

In FileSystemEntry, only refresh each cache when needed.

c4aa27f

Rename EnsureStatInitialized to EnsureCachesInitialized

371d1a4

Rename Refresh to RefreshCaches

f9e1619

Add ThrowOnCacheInitializationError

6c489ad

Only refresh relevant cache for IsHidden, IsReadOnly and IsSymbolicLink

136e2c3

carlossanlop added the area-System.IO label Jan 22, 2021

carlossanlop added this to the 6.0.0 milestone Jan 22, 2021

carlossanlop requested review from adamsitnik and Jozkee January 22, 2021 19:55

carlossanlop added this to In progress in Better for Linux via automation Jan 22, 2021

carlossanlop added this to In progress in System.IO - File system via automation Jan 22, 2021

carlossanlop self-assigned this Jan 22, 2021

carlossanlop marked this pull request as draft January 22, 2021 20:51

carlossanlop added 2 commits January 22, 2021 14:54

Ensure FileSystemEntry.IsHidden refreshes cache if needed. Invert Fil…

43476e7

…eSystemEntry.Initialize calls to refresh individual caches.

Add FileSystemEntry.HasHiddenPrefix to check for dot in name. Do soft…

a869d2c

… hidden check in Initialize to prevent perf regression.

danmoseley reviewed Jan 22, 2021

View reviewed changes

src/libraries/System.IO.FileSystem/tests/Enumeration/AttributeTests.cs Outdated Show resolved Hide resolved

Fix test comments

76dd793

carlossanlop marked this pull request as ready for review January 22, 2021 23:49

adamsitnik reviewed Jan 25, 2021

View reviewed changes

iSazonov reviewed Jan 25, 2021

View reviewed changes

iSazonov reviewed Jan 26, 2021

View reviewed changes

carlossanlop marked this pull request as draft January 29, 2021 17:42

iSazonov mentioned this pull request Feb 27, 2021

Proposed API for symbolic links #24271

Closed

Base automatically changed from master to main March 1, 2021 09:07

hamarb123 mentioned this pull request Mar 18, 2021

Fix setting creation date for OSX-like platforms #49555

Merged

ghost closed this Mar 31, 2021

Better for Linux automation moved this from In progress to Done Mar 31, 2021

System.IO - File system automation moved this from In progress to Done Mar 31, 2021

ghost locked as resolved and limited conversation to collaborators Apr 30, 2021

carlossanlop deleted the IsHidden2 branch February 7, 2024 06:42

This pull request was closed.

	public void SkippingHiddenFiles_Unix()
	public void SkippingFilesHiddenWithDotPrefixOnUnix()

	public void SkippingHiddenFiles_Windows_OSX()
	public void SkippingFilesHiddenWithAttributesOnWindowsAndOSX()

	new EnumerationOptions() { AttributesToSkip = 0 })
	new EnumerationOptions())

	entry._status = default;
	entry._status.Invalidate();
	entry._status = FileStatus.Create(entry, directoryEntry, out FileAttributes attributes);

	internal bool HasSecondaryDirectoryFlag => HasDirectoryFlag(_secondaryCache);
	internal bool IsSymlinkToDirectory => HasDirectoryFlag(_secondaryCache);

	(_initializedSecondaryCache == -1 \|\| _initializedSecondaryCache == 0); // Only refreshed when path is detected to be a symbolic link
	_initializedSecondaryCache <= 0; // Only refreshed when path is detected to be a symbolic link


		internal bool IsSecondaryCacheValid => _initializedSecondaryCache == 0;

		private bool IsValid =>

		}
		private void ThrowOnCacheInitializationError(ReadOnlySpan<char> path)

	internal bool InitiallyDirectory { get; set; }
	internal bool InitiallyDirectory { get; private set; }

Fix Unix FileStatus flag retrieval without perf regression #47348

Fix Unix FileStatus flag retrieval without perf regression #47348

Conversation

carlossanlop commented Jan 22, 2021 • edited Loading

stephentoub commented Jan 22, 2021

carlossanlop commented Jan 22, 2021

carlossanlop commented Jan 22, 2021 • edited Loading

adamsitnik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlossanlop Jan 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamsitnik Jan 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlossanlop Jan 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlossanlop commented Jan 29, 2021

iSazonov commented Feb 11, 2021

iSazonov commented Feb 25, 2021

ghost commented Mar 31, 2021

carlossanlop commented Jan 22, 2021 •

edited

Loading

carlossanlop commented Jan 22, 2021 •

edited

Loading

carlossanlop Jan 25, 2021 •

edited

Loading

adamsitnik Jan 26, 2021 •

edited

Loading

carlossanlop Jan 25, 2021 •

edited

Loading