Add package building tools #63

Tostti · 2023-05-30T19:59:06Z

Description

This PR aims to add the necessary tools to build rpm and deb packages.

Issues Resolved

#58

Check List

yenienserrano · 2023-06-09T08:43:11Z

Maybe you could add documentation on how to use it.

dev-tools/build-packages/base/builder.sh

Desvelao · 2023-06-19T14:53:26Z

I experienced the following problem:

I built:

Wazuh dashboard in tar.gz (from currunt wz-init branch)
Security plugin for Wazuh dashboard in .zip (from currunt wz-init branch)

./generate_base.sh --dashboard-path /tmp/linux-x64-2.4.1.zip --security-path /tmp/security-dashboards-2.4.1.0.zip --app-url https://packages-dev.wazuh.com/pre-release/ui/dashboard/wazuh-4.4.0-1.zip --version 4.4.0
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            https://docs.docker.com/go/buildx/

Sending build context to Docker daemon  106.8MB
Step 1/10 : FROM rockylinux:8.5
 ---> 210996f98b85
Step 2/10 : RUN yum clean all && yum update -y
 ---> Using cache
 ---> 69dbfda95993
Step 3/10 : RUN yum install -y     curl     tar     findutils     git     xz      gcc     make     bc     sed     gzip     autoconf     automake     libtool     python3-devel     python3-pip     jq     unzip
 ---> Using cache
 ---> 29dec080f368
Step 4/10 : RUN git clone https://github.com/google/brotli.git
 ---> Using cache
 ---> 007e02c64422
Step 5/10 : RUN cd brotli && chmod +x ./bootstrap && ./bootstrap && ./configure --prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib64/brotli --libdir=/usr/lib64/brotli --datarootdir=/usr/share --mandir=/usr/share/man/man1 --docdir=/usr/share/doc     && make && make install
 ---> Using cache
 ---> 5033c0b43f8c
Step 6/10 : ADD builder.sh /usr/local/bin/builder
 ---> Using cache
 ---> 5c03c888deec
Step 7/10 : ADD opensearch-dashboards.tar.gz /opt
 ---> Using cache
 ---> 22f900b17b8d
Step 8/10 : ADD security-dashboards.zip /opt
 ---> Using cache
 ---> d5e7dc29c913
Step 9/10 : RUN chmod +x /usr/local/bin/builder
 ---> Using cache
 ---> f84574423c8f
Step 10/10 : ENTRYPOINT ["/usr/local/bin/builder"]
 ---> Using cache
 ---> 43bf6b36ca2c
Successfully built 43bf6b36ca2c
Successfully tagged dashboard_base_builder:latest
/usr/local/bin/builder: line 60: cd: /opt/wazuh-dashboard-base: Not a directory

I think the problem is located at this line: https://github.com/wazuh/wazuh-dashboard/pull/63/files#diff-27b565659462834488d30c45b7280780dc69ce02667763dcfaa75743257c7f7dR59, the opensearch-dashboards* references to a file instead of a directory to be renamed. After that, it tries to change the renamed directory, but it is not a directory, it is the Wazuh dashboard (.tar.gz) that was renamed.

AlexRuiz7 · 2023-06-22T09:27:11Z

Having problems build a deb package:

Command used:

./build_package.sh \
    --build-base yes
    --security-path /home/alex/wazuh/packages/security-dashboards-2.4.1.0.zip \ 
    --app-url https://packages-dev.wazuh.com/pre-release/ui/dashboard/wazuh-4.4.0-1.zip \
    --version 4.4.0

Result:

mk-build-deps: warning:     debian/changelog(l1): version '-1' is invalid: upstream version cannot be empty
LINE: wazuh-dashboard (-1) unstable; urgency=low
dh_testdir
dh_testroot
dh_prep
dh_testdir
dh_testroot
dh_install
dh_installdocs
dh_installchangelogs
dh_installchangelogs: warning:     debian/changelog(l1): version 'unknown' is invalid: version number does not start with digit
LINE: wazuh-dashboard-build-deps (unknown) unstable; urgency=low
Can't locate object method "is_valid" via package "unknown" (perhaps you forgot to load "unknown"?) at /usr/share/perl5/Debian/Debhelper/Dh_Lib.pm line 1044.
make: *** [debian/rules:24: binary-indep] Error 255
Error in the build process: exit status 2
dpkg: error: cannot access archive 'wazuh-dashboard-build-deps_unknown_all.deb': No such file or directory
mk-build-deps: dpkg --unpack failed
zsh: command not found: --security-path
zsh: command not found: --app-url

Desvelao · 2023-06-22T11:17:11Z

I experienced the following problem:

I built:

Wazuh dashboard in tar.gz (from currunt wz-init branch)
Security plugin for Wazuh dashboard in .zip (from currunt wz-init branch)

./generate_base.sh --dashboard-path /tmp/linux-x64-2.4.1.zip --security-path /tmp/security-dashboards-2.4.1.0.zip --app-url https://packages-dev.wazuh.com/pre-release/ui/dashboard/wazuh-4.4.0-1.zip --version 4.4.0
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            https://docs.docker.com/go/buildx/

Sending build context to Docker daemon  106.8MB
Step 1/10 : FROM rockylinux:8.5
 ---> 210996f98b85
Step 2/10 : RUN yum clean all && yum update -y
 ---> Using cache
 ---> 69dbfda95993
Step 3/10 : RUN yum install -y     curl     tar     findutils     git     xz      gcc     make     bc     sed     gzip     autoconf     automake     libtool     python3-devel     python3-pip     jq     unzip
 ---> Using cache
 ---> 29dec080f368
Step 4/10 : RUN git clone https://github.com/google/brotli.git
 ---> Using cache
 ---> 007e02c64422
Step 5/10 : RUN cd brotli && chmod +x ./bootstrap && ./bootstrap && ./configure --prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib64/brotli --libdir=/usr/lib64/brotli --datarootdir=/usr/share --mandir=/usr/share/man/man1 --docdir=/usr/share/doc     && make && make install
 ---> Using cache
 ---> 5033c0b43f8c
Step 6/10 : ADD builder.sh /usr/local/bin/builder
 ---> Using cache
 ---> 5c03c888deec
Step 7/10 : ADD opensearch-dashboards.tar.gz /opt
 ---> Using cache
 ---> 22f900b17b8d
Step 8/10 : ADD security-dashboards.zip /opt
 ---> Using cache
 ---> d5e7dc29c913
Step 9/10 : RUN chmod +x /usr/local/bin/builder
 ---> Using cache
 ---> f84574423c8f
Step 10/10 : ENTRYPOINT ["/usr/local/bin/builder"]
 ---> Using cache
 ---> 43bf6b36ca2c
Successfully built 43bf6b36ca2c
Successfully tagged dashboard_base_builder:latest
/usr/local/bin/builder: line 60: cd: /opt/wazuh-dashboard-base: Not a directory

I think the problem is located at this line: https://github.com/wazuh/wazuh-dashboard/pull/63/files#diff-27b565659462834488d30c45b7280780dc69ce02667763dcfaa75743257c7f7dR59, the opensearch-dashboards* references to a file instead of a directory to be renamed. After that, it tries to change the renamed directory, but it is not a directory, it is the Wazuh dashboard (.tar.gz) that was renamed.

I commented on this with @Tostti , and he checked this is working.

I have reviewed and it seems was a problem by using a zipped Wazuh dashboard instead a .tar.gz

$ ./generate_base.sh --dashboard-path ~/wazuh/tmp/opensearch-dashboards-2.4.1-linux-x64.tar.gz --security-path ~/wazuh/tmp/security-dashboards-2.4.1.0.zip --app-url https://packages-dev.wazuh.com/pre-release/ui/dashboard/wazuh-4.4.0-1.zip --version 4.4.0
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            https://docs.docker.com/go/buildx/

Sending build context to Docker daemon  107.3MB
Step 1/10 : FROM rockylinux:8.5
 ---> 210996f98b85
Step 2/10 : RUN yum clean all && yum update -y
 ---> Using cache
 ---> 69dbfda95993
Step 3/10 : RUN yum install -y     curl     tar     findutils     git     xz      gcc     make     bc     sed     gzip     autoconf     automake     libtool     python3-devel     python3-pip     jq     unzip
 ---> Using cache
 ---> 29dec080f368
Step 4/10 : RUN git clone https://github.com/google/brotli.git
 ---> Using cache
 ---> 007e02c64422
Step 5/10 : RUN cd brotli && chmod +x ./bootstrap && ./bootstrap && ./configure --prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib64/brotli --libdir=/usr/lib64/brotli --datarootdir=/usr/share --mandir=/usr/share/man/man1 --docdir=/usr/share/doc     && make && make install
 ---> Using cache
 ---> 5033c0b43f8c
Step 6/10 : ADD builder.sh /usr/local/bin/builder
 ---> Using cache
 ---> 5c03c888deec
Step 7/10 : ADD opensearch-dashboards.tar.gz /opt
 ---> 869e577e8f7b
Step 8/10 : ADD security-dashboards.zip /opt
 ---> 3fa0125f8a3f
Step 9/10 : RUN chmod +x /usr/local/bin/builder
 ---> Running in 2e99e65a04de
Removing intermediate container 2e99e65a04de
 ---> 1b5871d37243
Step 10/10 : ENTRYPOINT ["/usr/local/bin/builder"]
 ---> Running in 691ac44f01af
Removing intermediate container 691ac44f01af
 ---> 170fe1bc6a35
Successfully built 170fe1bc6a35
Successfully tagged dashboard_base_builder:latest
Archive:  wazuh-4.4.0-1.zip
  inflating: opensearch-dashboards/wazuh/package.json  
Attempting to transfer from file:/opt/security-dashboards.zip
Transferring 1185795 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation complete
Base file wazuh-dashboard-base-4.4.0-1-linux-x64.tar.xz added to /tmp/src/wazuh-dashboard/dev-tools/build-packages/base/output.

The base package seems to be built correctly.

Tostti · 2023-06-22T11:47:21Z

Having problems build a deb package:

Command used:

./build_package.sh \
    --build-base yes
    --security-path /home/alex/wazuh/packages/security-dashboards-2.4.1.0.zip \ 
    --app-url https://packages-dev.wazuh.com/pre-release/ui/dashboard/wazuh-4.4.0-1.zip \
    --version 4.4.0

Result:

mk-build-deps: warning:     debian/changelog(l1): version '-1' is invalid: upstream version cannot be empty
LINE: wazuh-dashboard (-1) unstable; urgency=low
dh_testdir
dh_testroot
dh_prep
dh_testdir
dh_testroot
dh_install
dh_installdocs
dh_installchangelogs
dh_installchangelogs: warning:     debian/changelog(l1): version 'unknown' is invalid: version number does not start with digit
LINE: wazuh-dashboard-build-deps (unknown) unstable; urgency=low
Can't locate object method "is_valid" via package "unknown" (perhaps you forgot to load "unknown"?) at /usr/share/perl5/Debian/Debhelper/Dh_Lib.pm line 1044.
make: *** [debian/rules:24: binary-indep] Error 255
Error in the build process: exit status 2
dpkg: error: cannot access archive 'wazuh-dashboard-build-deps_unknown_all.deb': No such file or directory
mk-build-deps: dpkg --unpack failed
zsh: command not found: --security-path
zsh: command not found: --app-url

@AlexRuiz7 I think the error is that you are not specifying any dashboard-path nor dashboard-url. There is a verification but apparently is not working properly, will review it.

Desvelao · 2023-06-22T12:49:06Z

Tests

Test	Status
Generate `.deb` package	🟢
Generate `.rpm` package	🟢

Generate .deb package

dpkg-deb: building package 'wazuh-dashboard' in '../wazuh-dashboard_4.4.0-1_amd64.deb'.
 dpkg-genbuildinfo --build=binary
 dpkg-genchanges --build=binary >../wazuh-dashboard_4.4.0-1_amd64.changes
dpkg-genchanges: warning: the current version (4.4.0-1) is earlier than the previous one (4.4.2-1)
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)

WARNING generated by debuild:
Making debian/rules executable!

Package wazuh-dashboard_4.4.0-1_amd64.deb.sha512 added to /tmp/wazuh-dashboard/dev-tools/build-packages/deb/output.

Generate .rpm package

Requires(pre): /bin/sh /usr/sbin/groupadd /usr/sbin/useradd
Requires(post): /bin/sh
Requires(preun): /bin/sh /sbin/service
Requires(postun): /bin/sh /sbin/service
Requires(posttrans): /bin/sh
Checking for unpackaged file(s): /usr/lib/rpm/check-files /build/rpmbuild/BUILDROOT/wazuh-dashboard-4.4.0-1.x86_64
Wrote: /build/rpmbuild/SRPMS/wazuh-dashboard-4.4.0-1.src.rpm
Wrote: /build/rpmbuild/RPMS/x86_64/wazuh-dashboard-4.4.0-1.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.YzqolU
+ umask 022
+ cd /build/rpmbuild/BUILD
+ rm -fr /build/rpmbuild/BUILDROOT/wazuh-dashboard-4.4.0-1.x86_64
+ exit 0
Package wazuh-dashboard-4.4.0-1.x86_64.rpm.sha512 added to /tmp/wazuh-dashboard/dev-tools/build-packages/rpm/output.

AlexRuiz7 · 2023-07-06T11:45:39Z

Superseded by #71

Upload modified files

3e219a6

Tostti self-assigned this May 30, 2023

Tostti linked an issue May 30, 2023 that may be closed by this pull request

Research about the generation of OS packages (rpm, deb) #58

Closed

Tostti added 12 commits May 31, 2023 12:41

Add base modifications

4af350c

Add deb modifications

79532d2

Fix ambiguous shebangs

7659141

Add rpm modifications

7d4f05e

Add output to gitignore

14c645b

Remove unnecessary files

0692d3b

Remove unnecessary files

17e9138

Clean base code

5f4d512

Clean base code

93ed96b

Clean deb code

16bd44e

Clean deb code

6a2c238

Clean rpm code

29fe0d1

Tostti marked this pull request as ready for review June 1, 2023 15:56

yenienserrano reviewed Jun 12, 2023

View reviewed changes

dev-tools/build-packages/base/builder.sh Show resolved Hide resolved

AlexRuiz7 mentioned this pull request Jun 14, 2023

Upgrade to version 2.7.0 of OpenSearch Dashboards #65

Merged

8 tasks

Add logic to support local plugin

7a31f9f

Desvelao reviewed Jun 19, 2023

View reviewed changes

dev-tools/build-packages/base/builder.sh Show resolved Hide resolved

Add field verification to deb and rpm

210cc42

AlexRuiz7 closed this Jul 6, 2023

AlexRuiz7 deleted the 58-add-package-building-tools branch July 6, 2023 11:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add package building tools #63

Add package building tools #63

Tostti commented May 30, 2023 •

edited

Loading

yenienserrano commented Jun 9, 2023

Desvelao commented Jun 19, 2023

AlexRuiz7 commented Jun 22, 2023

Desvelao commented Jun 22, 2023

Tostti commented Jun 22, 2023 •

edited

Loading

Desvelao commented Jun 22, 2023 •

edited

Loading

AlexRuiz7 commented Jul 6, 2023

Add package building tools #63

Add package building tools #63

Conversation

Tostti commented May 30, 2023 • edited Loading

Description

Issues Resolved

Check List

yenienserrano commented Jun 9, 2023

Desvelao commented Jun 19, 2023

AlexRuiz7 commented Jun 22, 2023

Desvelao commented Jun 22, 2023

Tostti commented Jun 22, 2023 • edited Loading

Desvelao commented Jun 22, 2023 • edited Loading

Tests

AlexRuiz7 commented Jul 6, 2023

Tostti commented May 30, 2023 •

edited

Loading

Tostti commented Jun 22, 2023 •

edited

Loading

Desvelao commented Jun 22, 2023 •

edited

Loading