Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: move docker folder to container #1898

Merged
merged 4 commits into from
Apr 5, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -742,7 +742,7 @@ The Unix collector is unable to monitor pollers running in containers. See [#249

- New `include_contains` plugin allows you to select a subset of objects. e.g. selecting only volumes with custom-defined ONTAP metadata

- We've included more out-of-the-box [Prometheus alerts](https://github.com/NetApp/harvest/blob/main/docker/prometheus/alert_rules.yml). Keep sharing your most useful alerts!
- We've included more out-of-the-box [Prometheus alerts](https://github.com/NetApp/harvest/blob/main/container/prometheus/alert_rules.yml). Keep sharing your most useful alerts!

- 7mode workflows continue to be improved :heart: Harvest now collects Qtree and Quotas counters from 7mode filers (these are already collected in cDOT)

Expand Down Expand Up @@ -775,7 +775,7 @@ The Unix collector is unable to monitor pollers running in containers. See [#249

- Harvest should provide a `include_contains` label agent plugin for filtering #735 Thanks to @chadpruden for reporting

- Improve Harvest's container compatibility with K8s via kompose. [#655](https://github.com/NetApp/harvest/pull/655) See [also](https://github.com/NetApp/harvest/tree/main/docker/k8) and [discussion](https://github.com/NetApp/harvest/discussions/827)
- Improve Harvest's container compatibility with K8s via kompose. [#655](https://github.com/NetApp/harvest/pull/655) See [also](https://github.com/NetApp/harvest/tree/main/container/k8) and [discussion](https://github.com/NetApp/harvest/discussions/827)

- The ZAPI cli tool should include counter types when querying ZAPIs #663

Expand All @@ -786,13 +786,13 @@ The output of one plugin can be fed into the input of the next one. #736 Thanks

- Harvest should collect Antivirus counters when ONTAP offbox vscan is configured [#346](https://github.com/NetApp/harvest/issues/346) Thanks to @burkl and @Falcon667 for reporting

- [Document](https://github.com/NetApp/harvest/tree/main/docker/containerd) how to run Harvest with `containerd` and `Rancher`
- [Document](https://github.com/NetApp/harvest/tree/main/container/containerd) how to run Harvest with `containerd` and `Rancher`

- Qtree counters should be collected for 7-mode filers #766 Thanks to @jmg011 for raising this issue and iterating with us on a solution

- Harvest admin node should work with pollers running in Docker compose [#678](https://github.com/NetApp/harvest/pull/678)

- [Document](https://github.com/NetApp/harvest/tree/main/docker/podman) how to run Harvest with Podman. Several RHEL customers asked about this since Podman ships as the default container runtime on RHEL8+.
- [Document](https://github.com/NetApp/harvest/tree/main/container/podman) how to run Harvest with Podman. Several RHEL customers asked about this since Podman ships as the default container runtime on RHEL8+.

- Harvest should include a Systemd service file for the HTTP service discovery admin node [#656](https://github.com/NetApp/harvest/pull/656)

Expand Down Expand Up @@ -1150,7 +1150,7 @@ On RHEL and Debian, the example Unix collector does not work at the moment due t

- Improve client certificate authentication [documentation](https://github.com/NetApp/harvest/issues/314#issuecomment-882120238)

- Improve Docker deployment documentation [4019308](https://github.com/NetApp/harvest/tree/main/docker/onePollerPerContainer)
- Improve Docker deployment documentation [4019308](https://github.com/NetApp/harvest/tree/main/container/onePollerPerContainer)

### Fixes

Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ dist-tar:
@rm -rf ${DIST}
@mkdir ${TMP}
@mkdir ${DIST}
@cp -r .git cmd bin conf docker docs grafana pkg service cert autosupport go.mod go.sum Makefile README.md LICENSE prom-stack.tmpl harvest.cue ${TMP}
@cp -r .git cmd bin conf container docs grafana pkg service cert autosupport go.mod go.sum Makefile README.md LICENSE prom-stack.tmpl harvest.cue ${TMP}
@cp harvest.yml ${TMP}/harvest.yml
@tar --directory /tmp --create --gzip --file ${DIST}/${HARVEST_PACKAGE}.tar.gz ${HARVEST_PACKAGE}
@rm -rf ${TMP}
Expand Down
4 changes: 2 additions & 2 deletions cmd/tools/generate/generate.go
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ func generateDocker(path string, kind int) {
filesd = append(filesd, fmt.Sprintf("- targets: ['%s:%d']", pollerInfo.ServiceName, pollerInfo.Port))
}

t, err := template.New("docker-compose.tmpl").ParseFiles("docker/onePollerPerContainer/docker-compose.tmpl")
t, err := template.New("docker-compose.tmpl").ParseFiles("container/onePollerPerContainer/docker-compose.tmpl")
if err != nil {
panic(err)
}
Expand Down Expand Up @@ -404,7 +404,7 @@ func init() {

dFlags.BoolVarP(&opts.showPorts, "port", "p", false, "Expose poller ports to host machine")
_ = dockerCmd.MarkPersistentFlagRequired("output")
fFlags.StringVar(&opts.filesdPath, "filesdpath", "docker/prometheus/harvest_targets.yml",
fFlags.StringVar(&opts.filesdPath, "filesdpath", "container/prometheus/harvest_targets.yml",
"Prometheus file_sd target path. Written when the --output is set")
fFlags.IntVar(&opts.promPort, "promPort", 9090, "Prometheus Port")
fFlags.IntVar(&opts.grafanaPort, "grafanaPort", 3000, "Grafana Port")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,12 @@ After Rancher Desktop installs, start it `Cmd + Space` type: Rancher and wait fo

## Create Harvest Docker image

You only need to [create a new image](https://github.com/NetApp/harvest/tree/main/docker/onePollerPerContainer#building-harvest-docker-image) if you've made changes to Harvest. If you just want to use the latest version of Harvest, skip this step.
You only need to [create a new image](https://github.com/NetApp/harvest/tree/main/container/onePollerPerContainer#building-harvest-docker-image) if you've made changes to Harvest. If you just want to use the latest version of Harvest, skip this step.

These are the same steps outline on [Building Harvest Docker Image](https://github.com/NetApp/harvest/tree/main/docker/onePollerPerContainer#building-harvest-docker-image) except we replace `docker build` with `nerdctl` like so:
These are the same steps outline on [Building Harvest Docker Image](https://github.com/NetApp/harvest/tree/main/container/onePollerPerContainer#building-harvest-docker-image) except we replace `docker build` with `nerdctl` like so:

```sh
nerdctl build -f docker/onePollerPerContainer/Dockerfile -t harvest:latest . --no-cache
nerdctl build -f container/onePollerPerContainer/Dockerfile -t harvest:latest . --no-cache
```

## Generate a Harvest compose file
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ You only need to build a Harvest Docker image if you want a custom image. If ins
Build Docker Image

```
docker build -f docker/onePollerPerContainer/Dockerfile -t harvest:latest . --no-cache
docker build -f container/onePollerPerContainer/Dockerfile -t harvest:latest . --no-cache
or
nerdctl build -f docker/onePollerPerContainer/Dockerfile -t harvest:latest . --no-cache
nerdctl build -f container/onePollerPerContainer/Dockerfile -t harvest:latest . --no-cache
```
File renamed without changes.
2 changes: 1 addition & 1 deletion deb/build-deb.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ mkdir -p "$BUILD/opt/harvest/bin/"
cp -r "$SRC/grafana" "$SRC/conf" "$BUILD/opt/harvest/"
cp "$SRC/harvest.yml" "$SRC/prom-stack.tmpl" "$SRC/harvest.cue" "$BUILD/opt/harvest/"
cp -r "$SRC/.github/" "$BUILD/opt/harvest/"
cp -r "$SRC/pkg/" "$SRC/cmd/" "$SRC/docs/" "$SRC/docker/" "$BUILD/opt/harvest/"
cp -r "$SRC/pkg/" "$SRC/cmd/" "$SRC/docs/" "$SRC/container/" "$BUILD/opt/harvest/"
cp -r "$SRC/rpm/" "$SRC/deb/" "$SRC/service/" "$SRC/cert/" "$SRC/autosupport/" "$SRC/.git" "$BUILD/opt/harvest/"
cp "$SRC/Makefile" "$SRC/README.md" "$SRC/LICENSE" "$SRC/go.mod" "$SRC/go.sum" "$BUILD/opt/harvest/"
if [ -d "$SRC/vendor" ]; then
Expand Down
8 changes: 4 additions & 4 deletions docs/install/containers.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Harvest is container-ready and supports several deployment options:
- [Stand-up Prometheus, Grafana, and Harvest via Docker Compose](#docker-compose). Choose this if
you want to hit the ground running. Install, volume and network mounts automatically handled.

- [Poller-per-container model](https://github.com/NetApp/harvest/tree/main/docker/onePollerPerContainer) that offers
- [Poller-per-container model](https://github.com/NetApp/harvest/tree/main/container/onePollerPerContainer) that offers
more flexibility in configuration. This deployment enables a broad range of orchestrators (Nomad, Mesosphere, Swarm,
K8, etc.) since you pick-and-choose what gets built and how it's deployed, stronger familiarity with containers is
recommended.
Expand All @@ -15,10 +15,10 @@ Harvest is container-ready and supports several deployment options:
stands up Harvest, Grafana, and Prometheus.

- Want to run Harvest on a Mac
via [containerd and Racher Desktop](https://github.com/NetApp/harvest/tree/main/docker/containerd)? We got you
via [containerd and Racher Desktop](https://github.com/NetApp/harvest/tree/main/container/containerd)? We got you
covered.

- [Local K8 Deployment](https://github.com/NetApp/harvest/blob/main/docker/k8/README.md) via Kompose
- [Local K8 Deployment](https://github.com/NetApp/harvest/blob/main/container/k8/README.md) via Kompose

## Docker Compose

Expand Down Expand Up @@ -80,7 +80,7 @@ bin/harvest generate docker full --output harvest-compose.yml

1. Creates a Docker compose file with a container for each Harvest poller defined in your `harvest.yml`
2. Creates a matching Prometheus service discovery file for each Harvest poller (located
in `docker/prometheus/harvest_targets.yml`). Prometheus uses this file to scrape the Harvest pollers.
in `container/prometheus/harvest_targets.yml`). Prometheus uses this file to scrape the Harvest pollers.

### Start everything

Expand Down
4 changes: 2 additions & 2 deletions docs/prometheus-exporter.md
Original file line number Diff line number Diff line change
Expand Up @@ -303,8 +303,8 @@ Prometheus includes out-of-the-box support for simple alerting. Alert rules are
file. Setup and details can be found in the Prometheus
guide on [alerting](https://prometheus.io/docs/practices/alerting/).

Harvest also includes [ems alerts](https://github.com/NetApp/harvest/blob/main/docker/prometheus/ems_alert_rules.yml)
and [sample alerts](https://github.com/NetApp/harvest/blob/main/docker/prometheus/alert_rules.yml) for reference.
Harvest also includes [ems alerts](https://github.com/NetApp/harvest/blob/main/container/prometheus/ems_alert_rules.yml)
and [sample alerts](https://github.com/NetApp/harvest/blob/main/container/prometheus/alert_rules.yml) for reference.
Refer [EMS Collector](https://github.com/NetApp/harvest/blob/main/cmd/collectors/ems/README.md) for more details about
EMS events.

Expand Down
6 changes: 3 additions & 3 deletions integration/Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ pipeline {
echo "Installing"
export BUILD_PATH=$NATIVE
export VERSION=$VERSION
cp /u/mpeg/harvest/harvest_targets.yml $WORKSPACE/harvest/docker/prometheus/
cp /u/mpeg/harvest/harvest_targets.yml $WORKSPACE/harvest/container/prometheus/
bash $WORKSPACE/harvest/integration/test/test.sh install_native
'''
}
Expand Down Expand Up @@ -104,7 +104,7 @@ pipeline {
echo "Installing"
export BUILD_PATH=http://harvest-jenkins.rtp.openenglab.netapp.com:8080/job/harvest2_0/job/artifcats/lastSuccessfulBuild/artifact/harvest-21.11.0-1.x86_64.rpm
export VERSION=$VERSION
cp /u/mpeg/harvest/harvest_targets.yml $WORKSPACE/harvest/docker/prometheus/
cp /u/mpeg/harvest/harvest_targets.yml $WORKSPACE/harvest/container/prometheus/
bash $WORKSPACE/harvest/integration/test/test.sh install_rpm
'''
}
Expand Down Expand Up @@ -165,7 +165,7 @@ pipeline {
./bin/harvest stop
./bin/harvest generate docker full --port --output harvest-compose.yml
docker login -u $DOCKER_USER -p $DOCKER_PWD
docker build -f docker/onePollerPerContainer/Dockerfile -t rahulguptajss/harvest:latest -t cr.netapp.io/harvest:latest -t ghcr.io/netapp/harvest:latest . --no-cache --build-arg VERSION=$VERSION
docker build -f container/onePollerPerContainer/Dockerfile -t rahulguptajss/harvest:latest -t cr.netapp.io/harvest:latest -t ghcr.io/netapp/harvest:latest . --no-cache --build-arg VERSION=$VERSION
docker-compose -f prom-stack.yml -f harvest-compose.yml up -d --remove-orphans
sleep 1m
bash $WORKSPACE/harvest/integration/test/test.sh install_docker
Expand Down
2 changes: 1 addition & 1 deletion integration/test/installer/prometheus.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ func (p *Prometheus) Install() bool {
ipAddress := utils.GetOutboundIP()
cmd := exec.Command("docker", "run", "-d", "-p", utils.PrometheusPort+":"+utils.PrometheusPort,
"--add-host=localhost:"+ipAddress,
"-v", path+"/../../docker/prometheus/:/etc/prometheus/",
"-v", path+"/../../container/prometheus/:/etc/prometheus/",
p.image)
cmd.Stdout = os.Stdout
err := cmd.Start()
Expand Down
4 changes: 2 additions & 2 deletions jenkins/artifacts/jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ pipeline {
sh '''
targetLocation=$targetParentLocation$VERSION-$RELEASE-$BRANCH
docker login
docker build -f docker/onePollerPerContainer/Dockerfile --build-arg VERSION=$VERSION --build-arg RELEASE=$RELEASE --build-arg ASUP_MAKE_TARGET=$ASUP_MAKE_TARGET --build-arg GIT_TOKEN=$GIT_TOKEN -t ${imageName}:latest -t ${imageName}:$VERSION-$RELEASE -t ${jfrogImagePrefix}:latest -t ${jfrogImagePrefix}:$VERSION-$RELEASE -t ${ghcrImage}:latest -t ${ghcrImage}:$VERSION-$RELEASE . --no-cache
docker build -f container/onePollerPerContainer/Dockerfile --build-arg VERSION=$VERSION --build-arg RELEASE=$RELEASE --build-arg ASUP_MAKE_TARGET=$ASUP_MAKE_TARGET --build-arg GIT_TOKEN=$GIT_TOKEN -t ${imageName}:latest -t ${imageName}:$VERSION-$RELEASE -t ${jfrogImagePrefix}:latest -t ${jfrogImagePrefix}:$VERSION-$RELEASE -t ${ghcrImage}:latest -t ${ghcrImage}:$VERSION-$RELEASE . --no-cache
docker save -o ${targetLocation}/docker_harvest.tar ${imageName}:latest
'''
}
Expand Down Expand Up @@ -221,7 +221,7 @@ pipeline {
git push https://$GIT_TOKEN@github.com/NetApp/harvest.git --delete nightly
fi
/opt/home/gh_2.8.0_linux_386/bin/gh release create nightly $targetLocation/*.rpm $targetLocation/*.deb $targetLocation/*.gz --notes "Nightly builds may include bugs and other issues. You might want to use the stable releases instead." --title "Harvest Nightly Release" --prerelease --target main
docker build -f docker/onePollerPerContainer/Dockerfile --build-arg VERSION=$VERSION --build-arg RELEASE=$RELEASE --build-arg ASUP_MAKE_TARGET=$ASUP_MAKE_TARGET --build-arg GIT_TOKEN=$GIT_TOKEN -t ${imageName}:latest -t ${imageName}:nightly -t ${jfrogImagePrefix}:latest -t ${jfrogImagePrefix}:nightly -t ${ghcrImage}:latest -t ${ghcrImage}:nightly . --no-cache
docker build -f container/onePollerPerContainer/Dockerfile --build-arg VERSION=$VERSION --build-arg RELEASE=$RELEASE --build-arg ASUP_MAKE_TARGET=$ASUP_MAKE_TARGET --build-arg GIT_TOKEN=$GIT_TOKEN -t ${imageName}:latest -t ${imageName}:nightly -t ${jfrogImagePrefix}:latest -t ${jfrogImagePrefix}:nightly -t ${ghcrImage}:latest -t ${ghcrImage}:nightly . --no-cache
echo $GIT_TOKEN | docker login ghcr.io -u $GIT_USERNAME --password-stdin
docker push ${ghcrImage}:nightly
'''
Expand Down
2 changes: 1 addition & 1 deletion prom-stack.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ services:
container_name: prometheus
image: prom/prometheus:v2.33.1
volumes:
- ./docker/prometheus/:/etc/prometheus/
- ./container/prometheus/:/etc/prometheus/
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
Expand Down
2 changes: 1 addition & 1 deletion rpm/build-rpm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ mkdir -p "$BUILD/harvest/bin"
cp -r "$SRC/.git" "$BUILD/harvest"
cp -r "$SRC/cmd/" "$BUILD/harvest/"
cp -r "$SRC/conf/" "$BUILD/harvest/"
cp -r "$SRC/docker/" "$BUILD/harvest/"
cp -r "$SRC/container/" "$BUILD/harvest/"
cp -r "$SRC/docs/" "$BUILD/harvest/"
cp -r "$SRC/grafana/" "$BUILD/harvest/"
cp -r "$SRC/pkg/" "$BUILD/harvest/"
Expand Down