feat(api): Add npipe support #2018

olljanat · 2018-07-03T22:05:38Z

Updated PR for which replaces #1186 and solves merge issues on it.

Waiting for microsoft/go-winio#80 to be merged.

Closes #1179

Closes #1728

deviantony · 2018-07-04T09:28:14Z

Great, thanks @olljanat

seal-ss · 2018-07-04T12:49:45Z

Woot 🎉

olljanat · 2018-07-04T21:32:39Z

I published docker image for testing purposes which contains these changes . You can test it with command:

docker run -v \\.\pipe\docker_engine:\\.\pipe\docker_engine -p 9000:9000 -d ollijanatuinen/portainer:windows1803-amd64-npipe-1

Implementation is now done on way that when you add local Docker env you just need enable "Windows containers" switch on there:

NOTE! Currently that image it is build from my debug branch which also contains DialPipe workaround olljanat@0c564d7 and where I have rollbacked 61c285b as it looks to be broken (at least l always got "Unable to locate template file on disk" error with it).

Test results

Windows 10, Version 1803 + Docker 18.03.1-ce-win65 - works fine
Windows Server, Version 1803 + 18.03.1-ee-1 - gives me error:

2018/07/04 22:50:22 http error: Unable to proxy the request via the Docker socket (err=open //./pipe/docker_engine: Access is denied.) (code=500)

If I try without mapping npipe I got error like this:

2018/07/05 00:21:32 http error: Unable to proxy the request via the Docker socket (err=open //./pipe/docker_engine: The system cannot find the file specified.) (code=500

so it looks that npipe mapping works even on server version but I misses some permissions.

@StefanScherer any ideas what would be missing?

Other known issues currently:

If you add local connection without selecting Windows containers. It will be saved with incorrect settings even if you change it.
Swarm mode does not work correctly. Any access to services gives very weird errors.

deviantony · 2018-07-05T05:56:02Z

Great job @olljanat 👍

If you add local connection without selecting Windows containers. It will be saved with incorrect settings even if you change it.

An communication error should be raised as it is now when adding a local Unix environment when the socket file is not available.

StefanScherer · 2018-07-05T06:03:54Z

@olljanat Awesome!
The permission denied error may be related to non-admin user in NanoServer image.
Have you tried with this additional Dockerfile instruction?

USER ContainerAdministrator

I know Microsoft wants to deprecated this, but sometimes it's the only option.

deviantony · 2018-07-05T07:39:28Z

@olljanat

I have rollbacked 61c285b as it looks to be broken (at least l always got "Unable to locate template file on disk" error with it

Do you mind rebasing this PR on top of develop with #2024 to check if that template file issue is fixed?

olljanat · 2018-07-05T10:25:02Z

@StefanScherer yes "USER ContainerAdministrator" did the trick. Thanks.
@deviantony #2024 fixed template issue. Thanks

There is also now new version of Docker image ollijanatuinen/portainer:windows1803-amd64-npipe-2

Needs more testing but looks quite good.

seal-ss · 2018-07-05T11:39:43Z

@olljanat Ha, thanks good to know. I'll try your image soon. Do you encounter problems on 1803 that still requires the go-winio fixes?

olljanat · 2018-07-05T13:07:05Z

@seal-ss your DialPipe workaround is currently included to image just in case.

Only bigger issue which I can see now is that DockerHub authentication fails and that why image pull or service create features cannot be used but I'm investigating it.

olljanat · 2018-07-05T14:45:07Z

Docker pull and swarm features works on latest ollijanatuinen/portainer:windows1803-amd64-npipe-3 image.

@deviantony can you review? Only known issue left is that if user try use npipe on Linux it will generate http panic errors to log but I'm not sure what is correct way to fix them?

2018/07/05 13:49:50 http: panic serving 172.17.0.1:3744: runtime error: invalid memory address or nil pointer dereference
goroutine 31 [running]:
net/http.(*conn).serve.func1(0xc420429540)
        /usr/local/go/src/net/http/server.go:1697 +0xd0
panic(0xbdce00, 0x117ec70)
        /usr/local/go/src/runtime/panic.go:491 +0x283
github.com/portainer/portainer/http/proxy.(*proxyTransport).proxyDockerRequest(0x0, 0xc4202fa600, 0x0, 0xc42041e4c0, 0xc420069810)
        /go/src/github.com/portainer/portainer/http/proxy/docker_transport.go:66 +0x9a
github.com/portainer/portainer/http/proxy.(*localProxy).ServeHTTP(0xc420086038, 0x1138cc0, 0xc4202f4000, 0xc4202fa600)
        /go/src/github.com/portainer/portainer/http/proxy/local.go:22 +0x9d
net/http.StripPrefix.func1(0x1138cc0, 0xc4202f4000, 0xc4202fa500)
        /usr/local/go/src/net/http/server.go:1957 +0x1a5
net/http.HandlerFunc.ServeHTTP(0xc420420870, 0x1138cc0, 0xc4202f4000, 0xc4202fa500)
        /usr/local/go/src/net/http/server.go:1918 +0x44
github.com/portainer/portainer/http/handler/endpointproxy.(*Handler).proxyRequestsToDockerAPI(0xc420010d50, 0x1138cc0, 0xc4202f4000, 0xc4202fa500, 0                                                                                         x84739b)
        /go/src/github.com/portainer/portainer/http/handler/endpointproxy/proxy_docker.go:41 +0x480
github.com/portainer/portainer/http/handler/endpointproxy.(*Handler).(github.com/portainer/portainer/http/handler/endpointproxy.proxyRequestsToDocke                                                                                         rAPI)-fm(0x1138cc0, 0xc4202f4000, 0xc4202fa500, 0xc420420810)
        /go/src/github.com/portainer/portainer/http/handler/endpointproxy/handler.go:28 +0x48
github.com/portainer/portainer/http/error.LoggerHandler.ServeHTTP(0xc420141b90, 0x1138cc0, 0xc4202f4000, 0xc4202fa500)
        /go/src/github.com/portainer/portainer/http/error/error.go:24 +0x44
github.com/portainer/portainer/http/security.(*RequestBouncer).mwCheckAuthentication.func1(0x1138cc0, 0xc4202f4000, 0xc4202fa400)
        /go/src/github.com/portainer/portainer/http/security/bouncer.go:202 +0xd7
net/http.HandlerFunc.ServeHTTP(0xc4203b2960, 0x1138cc0, 0xc4202f4000, 0xc4202fa400)
        /usr/local/go/src/net/http/server.go:1918 +0x44
github.com/portainer/portainer/http/security.mwSecureHeaders.func1(0x1138cc0, 0xc4202f4000, 0xc4202fa400)
        /go/src/github.com/portainer/portainer/http/security/bouncer.go:119 +0xfd
net/http.HandlerFunc.ServeHTTP(0xc4203b2980, 0x1138cc0, 0xc4202f4000, 0xc4202fa400)
        /usr/local/go/src/net/http/server.go:1918 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc42017d340, 0x1138cc0, 0xc4202f4000, 0xc4202fa400)
        /go/src/github.com/gorilla/mux/mux.go:162 +0xed
net/http.StripPrefix.func1(0x1138cc0, 0xc4202f4000, 0xc4202fa000)
        /usr/local/go/src/net/http/server.go:1957 +0x1a5
net/http.HandlerFunc.ServeHTTP(0xc4204202d0, 0x1138cc0, 0xc4202f4000, 0xc4202fa000)
        /usr/local/go/src/net/http/server.go:1918 +0x44
github.com/portainer/portainer/http/handler.(*Handler).ServeHTTP(0xc4203c58c0, 0x1138cc0, 0xc4202f4000, 0xc4202fa000)
        /go/src/github.com/portainer/portainer/http/handler/handler.go:62 +0xacb
net/http.serverHandler.ServeHTTP(0xc4202340d0, 0x1138cc0, 0xc4202f4000, 0xc4202fa000)
        /usr/local/go/src/net/http/server.go:2619 +0xb4
net/http.(*conn).serve(0xc420429540, 0x11398c0, 0xc42018c3c0)
        /usr/local/go/src/net/http/server.go:1801 +0x71d
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2720 +0x288

deviantony · 2018-07-06T06:08:42Z

@olljanat I'll try to review this weekend and give you indications about the npipe on Linux issue.

olljanat · 2018-07-09T16:12:45Z

@deviantony ping

deviantony · 2018-07-09T20:05:20Z

Sorry @olljanat I'm a bit busy, I'll do a review tomorrow.

olljanat · 2018-07-09T20:37:52Z

No worries. I also found and fixed one new bug and did plenty of more testing.

There is now docker image ollijanatuinen/portainer:windows1803-amd64-npipe-4 build directly from this branch without DialPipe workaround and same version Windows binaries can be found from: https://github.com/olljanat/portainer/files/2177891/portainer-with-npipe-support-pre4.zip

@seal-ss I did quite lot of testing with this one I was not able to repeat that old issue easily. I did see that once on Win 10 but after restart it disappeared and I have not ever seen that with Windows Server, version 1803.

So IMO we can merge this even before DialPipe fix merged.

StefanScherer · 2018-07-10T07:01:13Z

@olljanat That looks awesome! I've started your image in a Windows Server 1803 swarm (but currently just a docker run -d)
Sometimes I see problems when I switch to the dashboard: Failure
Unable to proxy the request via the Docker socket
But all the other views (Container, Services, Images, ...) work without a problem.

Another approach may be to retry the request for the several dashboard API calls to make it more stable.

olljanat · 2018-07-10T07:31:39Z

True. It happens only time to time and I only see that on dashboard but it is still there.

Now there is also test version ollijanatuinen/portainer:winio-pr80 which contains content on this PR + proposed fix from winio.

With that one I was not able duplicate that issue anymore.

StefanScherer · 2018-07-10T07:39:43Z

@olljanat Yes, this image works stable even for the dashboard 🎉

deviantony

Heya @olljanat

I did a first review, have a look at my comments. One thing though, could you rebase the PR on the latest version of the develop branch ? It'll be easier to review it.

Thanks !

deviantony · 2018-07-10T19:25:43Z

api/http/handler/websocket/websocket_exec.go

@@ -167,19 +167,30 @@ func createDial(endpoint *portainer.Endpoint) (net.Conn, error) {
 	var host string


I think that we should default to url.Host here by assuming url.Scheme == tcp.

This simplifies the if/else statement below.

host := url.Host if url.Scheme == "unix" || url.Scheme = "npipe" { host = url.Path }

deviantony · 2018-07-10T19:40:11Z

api/http/proxy/local.go

@@ -1,6 +1,6 @@
 package proxy

-// unixSocketHandler represents a handler to proxy HTTP requests via a unix:// socket
+// represents a handler to proxy HTTP requests via a unix:// socket and npipe:// named pipes


You can actually remove that comment.

deviantony · 2018-07-10T19:44:04Z

api/http/proxy/manager.go

@@ -51,6 +51,9 @@ func (manager *Manager) createDockerProxy(endpointURL *url.URL, tlsConfig *porta
 		}
 		return manager.proxyFactory.newDockerHTTPProxy(endpointURL, false), nil
 	}
+	if endpointURL.Scheme == "npipe" {


I think that we can assume npipe:// or unix:// here depending on the platform, no need for that check.

Simply use:

return manager.proxyFactory.newLocalProxy(endpointURL.Path), nil

See my comment above regarding the local_unix.go and local_windows.go files.

deviantony · 2018-07-10T19:45:46Z

api/http/proxy/factory_windows.go

+	"github.com/Microsoft/go-winio"
+)
+
+func (factory *proxyFactory) newNamedPipeProxy(path string) http.Handler {


Should be renamed newLocalProxy.

deviantony · 2018-07-10T19:46:58Z

api/http/proxy/factory_windows.go

@@ -0,0 +1,33 @@
+// +build windows


File should be renamed local_windows.go.

deviantony · 2018-07-10T19:47:52Z

api/http/proxy/factory_linux.go

+	"net/http"
+)
+
+func (factory *proxyFactory) newNamedPipeProxy(path string) http.Handler {


This function should be renamed newLocalProxy and it's content should be the actual content of newDockerSocketProxy (relocate the function here and rename it).

deviantony · 2018-07-10T19:49:00Z

app/docker/components/datatables/containers-datatable/containersDatatable.html

@@ -9,6 +9,57 @@
          <span class="setting" ng-class="{ 'setting-active': $ctrl.state.displayTextFilter }" ng-click="$ctrl.updateDisplayTextFilter()" ng-if="$ctrl.showTextFilter">
            <i class="fa fa-search" aria-hidden="true"></i> Search
          </span>
+          <span class="setting" ng-class="{ 'setting-active': $ctrl.columnVisibility.state.open }" uib-dropdown dropdown-append-to-body auto-close="disabled" is-open="$ctrl.columnVisibility.state.open">


Could you rebase your PR on the latest develop branch? It will be easier to review if I can only focus on the changes related to this PR.

True. My bad.

deviantony · 2018-07-10T19:49:41Z

app/portainer/services/api/endpointService.js

@@ -54,10 +54,15 @@ function EndpointServiceFactory($q, Endpoints, FileUploadService) {
    return Endpoints.remove({id: endpointID}).$promise;
  };

-  service.createLocalEndpoint = function() {
+  service.createLocalEndpoint = function(Windows) {


Please rename the parameter to useNamedPipe.

Hmm. If we assume npipe:// or unix:/// based on OS then we can remove this parameter and whole Windows/Linux containers selection.

I will change it like that.

Yes, if Portainer is running on Windows, then creating a local endpoint should default to use npipe. If running on Linux, default to use unix socket.

This is totally doable in the backend as we can create platform specific files (see my comment about local_windows.go and local_linux.go above).

But the frontend is not aware of the platform where Portainer is running. that's why #1186 introduced a Local boolean parameter in the endpoint creation payload.

OK. I will re-check the original implementation

deviantony · 2018-07-10T19:52:27Z

build/windows/nanoserver/Dockerfile

@@ -1,8 +1,10 @@
-FROM microsoft/nanoserver
+ARG SOURCE_TAG=latest


What's the point of adding this ARG layer?

I needed that to be able build Windows Server, version 1803 compatible version on Win 10 (docker build . --build-arg SOURCE_TAG=1803) other why it defaults to Windows Server 2016 based version.

@seal-ss can I still use your rebase-docker-image tool to build a 1803 image? If so, I believe that this line is not required.

Probably. I was just not familiar with that tool. I can test and remove ARG layer if that works.

Yeah our build workflow is not public yet, I still have all the docs on my side. I can give you more details on Slack at https://portainer.io/slack/, ping me there

Yes, the rebase-docker-image tool still should work for your COPY deployment builds.

deviantony · 2018-07-10T19:52:55Z

build/windows/nanoserver/Dockerfile


-VOLUME C:\\data


Why did you choose to remove the default VOLUME statement?

I find out on Windows Server, version 1803 where it tested these images using Docker process isolation mode that if VOLUME statement is on Dockerfile Portainer does not see C:\Data folder and wants create that but Windows things that it is there and folder creating fails. That that fails whole Portainer start process. Without that line it works correctly.

@seal-ss are you aware of any issues using the VOLUME statement in Windows server 1803?

Hmm. I did now some more testing and found that USER ContainerAdministrator line actually fixes also this issue.
If that line is not on file and VOLUME statement is starting portainer gives error like this:
2018/07/11 10:30:14 mkdir C:\data: Cannot create a file when that file already exists.

So I will restore that line to file.

deviantony · 2018-07-13T05:52:36Z

@StefanScherer thanks for the update and the information, I was not sure about this.

Yes, the portainer images embeds the Docker binary in order to manage Swarm stacks.

@olljanat Compose stack management on a standalone Docker host still need to be tested, as I am not sure about the usage of npipe with libcompose.

olljanat · 2018-07-13T08:26:34Z

@deviantony yes. Both compose version 2 and swarm stack deployments works like same way like than on 1.18.1. No new bug generated by this changes but I found one Windows containers related bug which already exists on 1.18.1 + TCP (will create old issue from that one).

Named Pipe support for Windows containers have been there from day 1 as that it way how docker cli communicates with docker daemon. Feature which we use here is ability mount Named Pipe inside of container and that requires at least Windows 10/Server, version 1709 + Docker 17.09.

But older versions can still use old way to communicate over TCP 2375.

So this one is ready for merge when winio fix is merged.

deviantony · 2018-07-13T08:35:26Z

Great, let's wait for the merge of microsoft/go-winio#80 then.

ragaar · 2018-07-16T18:50:28Z

Just to confirm, Will the final command look something like this?
docker run -d -p 9000:9000 -u ContainerAdministrator --name portainer --restart always -v \\.\pipe\docker_engine:\\.\pipe\docker_engine portainer/portainer

olljanat · 2018-07-16T19:11:30Z

@ragaar -u parameter is not need because it is set inside of Dockerfile and you most probably want include data folder mount too like it is on http://portainer.readthedocs.io/en/stable/deployment.html#persist-portainer-data

so result will be

docker run -d -p 9000:9000 --name portainer --restart always -v \\.\pipe\docker_engine:\\.\pipe\docker_engine -v C:\ProgramData\Portainer:C:\data portainer/portainer

We hopefully get this one included to release which is scheduled to next week.

deviantony · 2018-07-18T19:42:12Z

build/windows/nanoserver/Dockerfile

@@ -1,5 +1,7 @@
 FROM microsoft/nanoserver

+USER ContainerAdministrator


Quick question, will this statement cause any issue with current windows images or 1709 ?

No. I just quickly tested that docker run -t --rm -u ContainerAdministrator microsoft/nanoserver:1709 cmd /c echo %USERNAME% command works just fine.

And on microsoft/nanoserver:sac2016 it actually does not change anything as ContainerAdministrator is default setting on that version (but also does not break anything).

Yes, still fine for 1709 and 1803 windows images.

Great, thanks guys

olljanat · 2018-07-18T20:35:31Z

I noticed that this PR was not fully compatible with #2033 so I added one more commit to fix that issue.

deviantony

LGTM

deviantony · 2018-07-20T05:53:14Z

microsoft/go-winio#80 has been merged, I'll have a look later today to merge this one.

deviantony · 2018-07-20T09:02:48Z

Thanks to all the people involved !

@olljanat @StefanScherer 👍

deviantony · 2018-07-31T13:32:17Z

@olljanat @StefanScherer one of our user reported the following:

Hi, I am trying to run Portainer on a Windows 10 (Also tried on Windows Server 2016) but I keep getting this message on both the docker environments:

docker : C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: container d20b96fb14e43e112d6a660a23e84853fd90ca602259933fcdabcf941a74789a encountered an error during CreateProcess: failure in a Windows system call: The user name or password is incorrect. (0x52e) extra info: {"CommandLine":"/portainer.exe","User":"ContainerAdministrator","WorkingDirectory":"C:\\","CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}. At line:1 char:1 
docker run -d -p 9000:9000 --name portainer -v C:\ProgramData\docker\ ... 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
CategoryInfo : NotSpecified: (C:\Program File...leSize":[0,0]}.:String) [], RemoteException 
FullyQualifiedErrorId : NativeCommandError

I am running this docker command:

docker volume create portainer_data 
docker run -d -p 9000:9000 --name portainer -v C:\ProgramData\docker\Containers\Portainer:C:\data portainer/portainer

I also tried with --no-auth but I still get the same error. Maybe you can help.

Any pointers on what might be the cause?

ragaar · 2018-07-31T13:49:43Z

@deviantony
Doesn't look like it's tied to Window's pipe-engine. So, you might create a new issue. Though, I think what you're encountering could be referenced in the documentation.

I'd simplify the command you're running down to minimum inputs required and scale up from there to figure out which command is causing the error.

Also, just for reference you defined a data volume but to use it your volume command would look more like this: -v portainer_data:drive:\path\in\container.

Full disclosure: I've not done much with volumes in Windows, so I can't confirm the direction of the \ from memory. But that problem, I know, is easy to find online :)

deviantony · 2018-07-31T13:51:47Z

I did not actually execute these commands, someone reported it to us via e-mail :-)

I do not have a Windows environment on which I can try to reproduce at the moment. So I'm asking for any help here.

ragaar · 2018-07-31T13:53:48Z

Ahhh! That makes more sense. I don't have Windows either! lol

olljanat · 2018-07-31T13:56:32Z

@deviantony I just tested on Win 10 that both Linux and Windows containers works with commands given on here http://portainer.readthedocs.io/en/stable/deployment.html#windows except that there is small typo which I fixed on portainer/portainer-docs#54

Windows Server 2016 need still follow old documentation: http://portainer.readthedocs.io/en/stable/faq.html#how-can-i-setup-portainer-on-windows-server-2016

deviantony · 2018-07-31T13:57:54Z

@olljanat as you can see the user is simply running:

docker run -d -p 9000:9000 --name portainer -v C:\ProgramData\docker\Containers\Portainer:C:\data portainer/portainer

Which should be working on Win server 2016

olljanat · 2018-07-31T14:00:50Z

Yep but npipe is not used until user selects local connection.

So ask him open new issue with all details.

deviantony · 2018-07-31T14:06:14Z

Ok, I'll ask him to open an issue. Thanks @olljanat

* apply style to authentication logs and activity logs

deviantony mentioned this pull request Jul 4, 2018

Add npipe support #1186

Closed

deviantony added the work-in-progress Indicates that the PR is a work-in-progress and not ready yet label Jul 4, 2018

deviantony mentioned this pull request Jul 5, 2018

fix(cli): fix default template file path on Windows #2024

Merged

olljanat changed the title ~~WIP: feat(api): Add npipe support~~ feat(api): Add npipe support Jul 5, 2018

olljanat mentioned this pull request Jul 8, 2018

Support for Windows named pipes mounts moby/swarmkit#2691

Merged

This was referenced Jul 9, 2018

[Feature Request] Windows agent support #1929

Closed

Is there a roadmap for docker swarm? moby/swarmkit#2665

Open

olljanat force-pushed the add-npipe-support-merge branch from d77df8a to 1fead33 Compare July 10, 2018 15:10

deviantony added contrib/tech-review-in-progress and removed work-in-progress Indicates that the PR is a work-in-progress and not ready yet labels Jul 10, 2018

deviantony requested changes Jul 10, 2018

View reviewed changes

olljanat force-pushed the add-npipe-support-merge branch from 1fead33 to 92a16ad Compare July 10, 2018 21:05

deviantony added this to the 1.18.x milestone Jul 18, 2018

deviantony reviewed Jul 18, 2018

View reviewed changes

fix(api): home view snapshots with npipe

f000280

olljanat force-pushed the add-npipe-support-merge branch from 986fb66 to f000280 Compare July 18, 2018 20:33

deviantony approved these changes Jul 19, 2018

View reviewed changes

deviantony removed the status/pending label Jul 20, 2018

deviantony added contrib/ready-to-merge labels Jul 20, 2018

deviantony merged commit 4129550 into portainer:develop Jul 20, 2018

deviantony mentioned this pull request Jul 23, 2018

fix(api): fix invalid platform build statements #2064

Merged

deviantony modified the milestones: 1.18.x, 1.19.0 Jul 27, 2018

olljanat mentioned this pull request Oct 24, 2018

CPU and Memory Graphs not working for Windows Containers #1368

Closed

xAt0mZ pushed a commit that referenced this pull request Aug 25, 2022

feat(ui): ui change for authentication and activity logs EE-3798 (#2018)

87f36c8

* apply style to authentication logs and activity logs

		@@ -167,19 +167,30 @@ func createDial(endpoint *portainer.Endpoint) (net.Conn, error) {
		var host string

		@@ -1,8 +1,10 @@
		FROM microsoft/nanoserver
		ARG SOURCE_TAG=latest

		@@ -1,5 +1,7 @@
		FROM microsoft/nanoserver

		USER ContainerAdministrator

feat(api): Add npipe support #2018

feat(api): Add npipe support #2018

Conversation

olljanat commented Jul 3, 2018 • edited by deviantony

deviantony commented Jul 4, 2018

seal-ss commented Jul 4, 2018

olljanat commented Jul 4, 2018

deviantony commented Jul 5, 2018

StefanScherer commented Jul 5, 2018

deviantony commented Jul 5, 2018

olljanat commented Jul 5, 2018

seal-ss commented Jul 5, 2018 • edited

olljanat commented Jul 5, 2018

olljanat commented Jul 5, 2018

deviantony commented Jul 6, 2018

olljanat commented Jul 9, 2018

deviantony commented Jul 9, 2018

olljanat commented Jul 9, 2018

StefanScherer commented Jul 10, 2018

olljanat commented Jul 10, 2018

StefanScherer commented Jul 10, 2018

deviantony left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deviantony commented Jul 13, 2018 • edited

olljanat commented Jul 13, 2018

deviantony commented Jul 13, 2018

ragaar commented Jul 16, 2018

olljanat commented Jul 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olljanat commented Jul 18, 2018

deviantony left a comment

Choose a reason for hiding this comment

deviantony commented Jul 20, 2018

deviantony commented Jul 20, 2018

deviantony commented Jul 31, 2018

ragaar commented Jul 31, 2018 • edited

deviantony commented Jul 31, 2018

ragaar commented Jul 31, 2018

olljanat commented Jul 31, 2018

deviantony commented Jul 31, 2018

olljanat commented Jul 31, 2018

deviantony commented Jul 31, 2018

olljanat commented Jul 3, 2018 •

edited by deviantony

seal-ss commented Jul 5, 2018 •

edited

deviantony commented Jul 13, 2018 •

edited

ragaar commented Jul 31, 2018 •

edited