Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No resource type out-of-the-box with windows worker #533

Closed
iar-wain opened this issue Jul 13, 2016 · 16 comments

Comments

@iar-wain
Copy link

commented Jul 13, 2016

Hi,

According to http://concourse.ci/resource-types.html, Concourse workers should support resource types such as git out of the box.

I probably missed something obvious somewhere but it looks like it might not the case for the Windows worker (everything went fine on Linux).

Both server and worker were started on a single Win7 SP1 64 machine with the following command lines:

Server:
D:\>concourse web --development-mode --external-url http://127.0.0.1:8080 --session-signing-key session_signing_key --tsa-host-key host_key --tsa-authorized-keys authorized_worker_keys

Worker:
D:\>concourse worker --work-dir worker --tsa-host 127.0.0.1 --tsa-public-key host_key.pub --tsa-worker-private-key worker_key --peer-ip 127.0.0.1

And the following pipeline was used:

---
resources:
- name: git-res
  type: git
  source:
    uri: https://github.com/concourse/concourse.git

jobs:
- name: hello-world
  plan:
  - get: git-res
  - task: hello-world
    config:
      platform: windows
      run:
        path: echo
        args: [hello world]

When executing the hello-world job, an error message was obtained:

error
no workers satisfying: resource type 'git'

available workers: 
  - platform 'windows'

What did I miss?

Additionally I couldn't find any documentation on how to add custom resource types for Windows workers. Where should I look for that?

Thanks in advance.

@concourse-bot

This comment has been minimized.

Copy link

commented Jul 13, 2016

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

  • #126416531 No resource type out-of-the-box with windows worker

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

@vito

This comment has been minimized.

Copy link
Member

commented Jul 14, 2016

Resources are implemented as Docker images, so there aren't any for Windows yet. Someone could start porting them, though it doesn't sound very fun. At the moment you'll need at least one Linux worker to use resources. The bits will be fetched on Linux and streamed over to Windows.

@iar-wain

This comment has been minimized.

Copy link
Author

commented Jul 14, 2016

Thanks for the quick reply.

What would be my options at the moment for a build process that needs to run on Windows and that has some heavy dependencies (think of SDKs that take 1-2h + reboot to get installed)?

Is there a more convenient option beside:

  • having all the dependencies installed beforehand on the Windows worker
  • transgress the containers separation and have source/asset repositories ready outside of the containers (200+ GiB), that would only need to get synced by a task to get newer data
  • use a single job that contains all the tasks and which gets triggered via the REST API and/or fly
  • write the final output (40+ GiB) outside the containers (using an absolute path?)

Sorry if the questions sound silly, I'm quite new to that whole container thing. =)

Thanks in advance!

@benmoss

This comment has been minimized.

Copy link

commented Aug 27, 2016

@iar-wain on our builds we're installing MSIs that mess with the OS / and are stateful, so our scripts are made to detect whether dependencies are there and install them if not. here's an example: https://github.com/cloudfoundry/bosh-agent/blob/abc79750898ebc2aca42916e44f4bc3d2e395b45/ci/tasks/test-unit.ps1

your problem is a lil more complicated due to the size of the dependencies and the restart requirement, so i'd probably recommend number one, though if the dependencies are changing frequently that's gonna be more painful. hard to recommend without knowing the full scope of the problem, but there's not going to be any magic bullet here.

@petemounce

This comment has been minimized.

Copy link

commented Dec 22, 2016

Given Windows server 2016 is now available with docker support, is this still current? What is required to port the resources to Windows docker containers?

@occelebi

This comment has been minimized.

Copy link

commented Mar 6, 2017

@vito It does not even work when I have linux worker available.

$ fly -t backend watch --build 24
initializing
no workers satisfying: resource type 'docker-image', platform 'windows'

available workers: 
  - platform 'linux'
  - platform 'windows'
errored

- name: windows
  plan:
  - task: hello
    config:
      platform: windows
      image_resource:
        type: docker-image
        source: {repository: microsoft/dotnet}
      run:
        path: echo
        args: ['Hello, world!']
@benmoss

This comment has been minimized.

Copy link

commented Mar 7, 2017

@occelebi you would have to port the docker image resource to work on Windows

@occelebi

This comment has been minimized.

Copy link

commented Mar 7, 2017

@benmoss, @Fydon mention that on slack.
you can use containers on Windows, but you need to set up docker to use Windows containers, which is only available from stable version 1.13.

@marco-m

This comment has been minimized.

Copy link
Contributor

commented May 15, 2017

I am really confused both by this issue and by #1158. Please consider increasing the priority for documenting in general how to use concourse with Windows and Mac workers. thanks :-)

@mxplusb

This comment has been minimized.

Copy link

commented May 19, 2017

@occelebi so to consume Docker resources on a Windows VM, we need to set up Docker on the host, correct?

@occelebi

This comment has been minimized.

Copy link

commented May 20, 2017

@mxplusb yes as far as I know you need to set up docker on the host. However as vito mentioned

At the moment you'll need at least one Linux worker to use resources. The bits will be fetched on Linux and streamed over to Windows.

So if you want to fetch some image from any docker registry you would need at least one linux worker to download that image and stream over to Windows host. Then you can run that image on Windows as container. I highly doubt if that would be an efficient solution run windows image on windows host as windows images are only around for a while and they are very large in size.

@jtarchie

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2017

There some confusion here on how resources work.

@occelebi, the docker image for linux will never be streamed over to a windows worker. The resource will always run on the linux worker, only the output of the resource will be streamed to windows worker. For example, the git resource does a git clone into a directory, that directory will be stream to the worker.

The docker image used for resources are linux specific and require a linux worker. A windows worker running docker does not fulfill this role. The concourse linux workers actually don't user docker to run containers, it uses garden-runc, which supports docker images.

@occelebi your example above, will not work. When you specified platform: windows, concourse is looking for the resource docker-image on a windows worker, which concourse does not support.

@jtarchie jtarchie closed this Aug 29, 2017

@peimanja

This comment has been minimized.

Copy link

commented Jan 14, 2019

@jtarchie So I have a windows worker now. How can I run jobs on containers on that worker? I do not want to run jobs on the worker itself so is there an equivalent resource to docker-image on Windows?

@marco-m

This comment has been minimized.

Copy link
Contributor

commented Jan 14, 2019

@peimanja You cannot. For the time being, only Linux supports containers. All other platforms (Windows, macOS) will run tasks directly on the host.

@esauser

This comment has been minimized.

Copy link

commented May 6, 2019

For those following this thread, here is how we're handling containers on Windows.

@marco-m

This comment has been minimized.

Copy link
Contributor

commented May 6, 2019

@esauser I am flabbergasted! Wow! Thanks for the hint!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.