Docker dev environment support

[ashwin-pc]

Is your feature request related to a problem? Please describe.

Currently, development for OSD is optimized for the Linux environment. This adds friction to the development flow even if tools like remote dev servers are used. E.g. functional testing.

Describe the solution you'd like

Add a docker dev container so that every contributor can work on an identical dev environment and not worry about local OS limitations. This can additionally simplify the process of development with a bundled version of OpenSeach and plugins by simply switching out the dev container used.

Describe alternatives you've considered

Rolling out supported distributions for every major platform available.

Additional context

Add any other context or screenshots about the feature request here.

[joshuarrrr]

@ashwin-pc +1 this sounds great. Can you provide a few more concrete details about what you envision the docker startup and dev process to look like (rough pseudo-flow comparison to the current local dev would be fine).

[ashwin-pc]

Havent had the time to setup a POC for this yet but it should be fairly simple. We can have a base docker image that has all the dependencies we need to build OpenSearch Dashboards and build it and then use a compose file like this:

verison: "3.9"
services:
  os:
    image: opensearchproject/opensearch:latest
    ports:
      - "9300:9300"
    # Other env params
  osd:
    build: . # Or alternatively it can be an base image with all the dependencies installed that we can provide
    ports:
      - "5601:5601"
    volumes:
      - "./:{container_path_to_project_root}/"

to setup any docker image for OpenSearch Engine. Now we can connect to the osd container to run any commands that would not be supported on the native OS and any changes made to the code during development would run in the docker environment

[kristenTian]

[Triage]: @ashwin-pc is this issue sth you will be interested to look into?

[ashwin-pc]

I cant promise that i'll get around to it soon, but if no one picks this up in a month or two, I may give it a whirl.

[BSFishy]

Alternatively, Vagrant might be of use here, which can use Docker as a back-end and is made specifically with development in mind. It essentially spins up identical virtual machines (or Docker containers) for development.

[seraphjiang]

Alternatively, Vagrant might be of use here, which can use Docker as a back-end and is made specifically with development in mind. It essentially spins up identical virtual machines (or Docker containers) for development.

wow, never used that, could we get a demo :)

[ashwin-pc]

@BSFishy Thats actually pretty neat! Last time i used Vagrant, it could only run on a VM which was a big overhead especially if you are trying to develop on not the latest and greatest hardware. That being said i'd still lean just a lil bit towards only using docker since:

1. It is one less dependency
2. Looking at the docs it has for docker, I dont see a significant value add over using just docker
3. Even in their Vagrant vs Docker doc, they acknowledge that if you dont hit the limitations of docker (which we dont) a docker based workflow might actually be better
4. Vagrant (caveat: the last time i used it was around 3 years ago) was quite a lot harder to setup and get up and running and was honestly a lot more pain than good for development. This can be taken with a grain of salt since it seems to have matured since I last worked with it.

But I agree with @seraphjiang, it would be cool to see a working demo of this workflow!

[BSFishy]

I'll be working on a work-in-progress Vagrantfile over here. Right now, it's getting hung up on ssh access, but that may just be a configuration issue on my machine.

Additionally, ssh access might not even be necessary if I can configure it to just start the development server, then let OSD optimizer deal with file watching. It wouldn't be ideal for letting people run commands on the code base other than running the dev server, but I think it would be fine for a quick proof-of-concept.

[BSFishy]

After a bit more tampering, it seems that we would need a custom Docker image for SSH access. Essentially Vagrant needs the container to have sshd running, which most images, such as node do not. There are some images on Docker Hub, I think, but they mostly don't have what we need in an image, like Node.

I could try creating a new image with the necessary parts, but I'm not extremely experienced with creating Docker images, and at that point it may be more to maintain than just having a Docker configuration like what was previously posted here.

[abbyhu2000]

The docker setup is accomplished in this PR:#3444

Here is a YouTube demo on the Docker set up: https://www.youtube.com/watch?v=tEBExO9mMuU

Here is the Docker manual: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/docs/docker-dev/docker-dev-setup-manual.md