[8.0.0] Provide a release tarball with pre-generated npm stuff

[tgurr]

While trying to update our distribution packages I've come across the issue that gsa tries to download npm stuff during compile time which is quite unfortunate for us as a source-based distribution. Other projects using npm e.g. https://github.com/cockpit-project/cockpit and https://pypi.org/project/Mopidy-Iris/ come with precompiled npm stuff, it would be great if you could provide a complete release tarball as well.

[bjoernricks]

Personally I don't think including the created build files or including all the npm dependencies in the tarball is a good idea. But I am open for suggestions and a PR.

[hsntgm]

@bjoernricks

After executing yarn install in "gsa" folder it creates "node_modules" folder.If you compress this folder with tar as node_modules.tar.gz it just takes 67M.

It seems sharing that with us will be enough to compile "gsa" for source-based distribution with a little cmake hack.
Please let me know If you don't think to provide that modules pack so i will find an other way.

Thanks.

[bjoernricks]

I really would like to know how this is solved by other projects.

[cheese1]

i tried to do that while creating packages for fedora. nb.: the packages are not yet finished (the upgrade-path of the packages is still missing), but they seem to work for me.
i created an offline-cache and use that during building of the binary packages.
http://www.nosuchhost.net/~cheese/fedora/packages/29/SRPMS/openvas-gsa-8.0.0-1.fc29.src.rpm

[cfi-gb]

@tgurr @cheese1 Off-topic and just a FYI that there is some discussion on a consistent naming of the packages across all distributions ongoing on greenbone/gvm-libs#197

e.g. openvas-gsa is most likely even more confusing and there should only an openvas-scanner package now that the project name is "Greenbone Vulnerability Management (GVM)".

[cheese1]

Thanks I was in the middle of collecting infos how to change names of packages in fedora. Yours Josef

------------------------- VON: cfi-gb <notifications@github.com> GESENDET: Samstag, 20. April 2019 16:35 AN: gsa@noreply.github.com CC: cheese1; Mention BETREFF: Re: [greenbone/gsa] [8.0.0] Provide a release tarball with pre-generated npm stuff (#1261)

@tgurr[1] @cheese1[2] Off-topic and just a FYI that there is some discussion on a consistent naming of the packages across all distributions ongoing on greenbone/gvm-libs#197[3] e.g. openvas-gsa is most likely even more confusing and there should only an openvas-scanner package now that the project name is "Greenbone Vulnerability Management (GVM)". — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub[4], or mute the

thread[5].

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/greenbone/gsa","title":"greenbone/gsa","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/greenbone/gsa"}},"updates":{"snippets":[{"icon":"PERSON","message":"@cfi-gb in #1261: @tgurr @cheese1 Off-topic and just a FYI that there is some discussion on a consistent naming of the packages across all distributions ongoing on https://github.com/greenbone/gvm-libs/issues/197\r\n\r\ne.g. openvas-gsa is most likely even more confusing and there should only an openvas-scanner package now that the project name is \"Greenbone Vulnerability Management (GVM)\"."}],"action":{"name":"View Issue","url":"#1261 (comment)"}}} [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "#1261 (comment)", "url": "#1261 (comment)", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } }

]

Links: ------ [1] https://github.com/tgurr [2] https://github.com/cheese1 [3] greenbone/gvm-libs#197 [4] #1261 (comment) [5] https://github.com/notifications/unsubscribe-auth/ABRDWXQSNK5RV6XATN53W7TPRMS4BANCNFSM4HEKKCRA

[bjoernricks]

I really would like to know how this is solved by other projects.

Does anybody know how this issue is solved by others? I am not sure if yarn creates a reproducible node_modules directory and if it is worth to put it into a tar.gz.

[bjoernricks]

Again I really would like to fix this issue for a 8.0.1 release but I rely on downstream for a sufficient approach. Packaging my node_modules directory doesn't seem right for me.

[hsntgm]

@bjoernricks

There are two solutions.

1-) For 8.0.1 maybe you could consider to release offline tarball too with pre-generated node_modules in gsa folder.

You can find patches below for this setup.

disabling fetch phase of yarn in CMakeLists.txt and set NODE_ENV in package.json that points to path react-scripts.js in node_modules folder (in offline mode it can't find automatically).

I confirm that this setup works for compiling gsa for source based distros if we have pre-generated node modules in gsa directory.

2-) If you looking for other solution.The only way is sharing the all node_modules links that gsa needs to compile.In package.json there are ~40 node dependency but it seems they have also other dependencies too.I see them in yarn.lock file.This is the hard way for both of us.

greenbone-security-assistant-8.0.0-react-env.patch

--- a/gsa/package.json  2019-04-05 17:16:01.000000000 +0300
+++ b/gsa/package.json  2019-04-12 04:57:40.962420279 +0300
@@ -66,12 +66,12 @@
     "x2js": "^3.2.6"
   },
   "scripts": {
-    "test": "NODE_PATH=src react-scripts test",
-    "test:coverage": "NODE_PATH=src react-scripts test --coverage --maxWorkers 2",
+    "test": "NODE_PATH=src NODE_ENV=production node_modules/react-scripts/bin/react-scripts.js test",
+    "test:coverage": "NODE_PATH=src NODE_ENV=production node_modules/react-scripts/bin/react-scripts.js test --coverage --maxWorkers 2",
     "lint": "NODE_PATH=src eslint --max-warnings 0 src",
-    "start": "NODE_PATH=src react-scripts start",
-    "build": "NODE_PATH=src react-scripts build",
-    "eject": "NODE_PATH=src react-scripts eject"
+    "start": "NODE_PATH=src NODE_ENV=production node_modules/react-scripts/bin/react-scripts.js start",
+    "build": "NODE_PATH=src NODE_ENV=production node_modules/react-scripts/bin/react-scripts.js build",
+    "eject": "NODE_PATH=src NODE_ENV=production node_modules/react-scripts/bin/react-scripts.js eject"
   },
   "devDependencies": {
     "eslint-config-prettier": "^4.0.0",

greenbone-security-assistant-8.0.0-yarn-install.patch

--- a/gsa/CMakeLists.txt        2019-04-05 17:16:01.000000000 +0300
+++ b/gsa/CMakeLists.txt        2019-04-12 15:35:03.428038344 +0300
@@ -1091,28 +1091,17 @@
      ${GSA_JS_SRC_FILES}
      PARENT_SCOPE)
 
-
-add_custom_command (OUTPUT node-modules.stamp
-                    DEPENDS ${GSA_PKG_FILES}
-                    COMMAND ${INSTALLER} "${INSTALLER_ARGS}" "install"
-                    COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/node-modules.stamp
-                    WORKING_DIRECTORY ${GSA_SRC_DIR}
-                    COMMENT "Install gsa-ng js dependencies")
-
-
 add_custom_command (OUTPUT bundle.stamp
                     COMMAND ${INSTALLER} "${INSTALLER_ARGS}" run build
                     COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/bundle.stamp
                     WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
-                    DEPENDS  ${CMAKE_CURRENT_BINARY_DIR}/node-modules.stamp
-                             ${GSA_PKG_FILES}
+                    DEPENDS  ${GSA_PKG_FILES}
                              ${GSA_STATIC_SRC_FILES}
                              ${GSA_JS_SRC_FILES}
                     COMMENT "Build gsa-ng install files")
 
 add_custom_target (gsa-ng ALL
-                   DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/node-modules.stamp
-                           ${CMAKE_CURRENT_BINARY_DIR}/bundle.stamp)
+                   DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/bundle.stamp)
 
 set (GSA_STATIC_BUILD_FILES
      ${GSA_BUILD_DIR}/index.html)

[hsntgm]

ping for issue @bjoernricks -->

I have just checked repology.org and after 3 months only alpine linux bumped new gvm-libs and all other new gvm components.I believe that if you fix this npm/yarn issue maintainers will be able to bump new gvm-libs & components to linux repositories.

Hasan.

[bjoernricks]

Sorry I can't spent much time on this issue at the moment. My role at Greenbone changed a lot in the past months.

@hsntgm if I understand you correctly it would be enough to release the node_modules directory as a tarball too? Otherwise I am depending on downstream to provide a PR.

[hsntgm]

Congrats. for new role @bjoernricks
Yes it is enough node_modules directory as a tarball.

Thanks.

[bjoernricks]

@hsntgm thanks!

I've uploaded https://github.com/greenbone/gsa/releases/download/v8.0.0/gsa-node-modules-8.0.0.tar.gz Please check if that file fits for you or something is still missing.

[hsntgm]

Thank you @bjoernricks

This issue is solved.

I successfully compiled gsa on Gentoo with gsa-node-modules-8.0.0.tar.gz
In short, this will work for all source-based distros with below patches:

greenbone-security-assistant-8.0.0-react-env.patch
greenbone-security-assistant-8.0.0-yarn-install.patch

Other source-based distro maintainers could get the necessarry patches from my gvm-10 repo -->

https://github.com/hsntgm/gvm-10-gentoo-overlay/tree/master/net-analyzer/greenbone-security-assistant/files

After greenbone release gsa-8.0.1 (there are lots of bug fixing) i will push gvm to main gentoo tree.

[bjoernricks]

I'll try to get a new release for all GVM components next week.

So I am closing this issue for now. Feel free to reopen if something is still missing.

[tgurr]

@hsntgm

Other source-based distro maintainers could get the necessarry patches from my gvm-10 repo -->

https://github.com/hsntgm/gvm-10-gentoo-overlay/tree/master/net-analyzer/greenbone-security-assistant/files

Error 404, also couldn't find the files in question when looking at your available GitHub repositories.

@bjoernricks First thanks for taking care of providing a tarball with the pre-generated files! However I don't really consider this fixed as long as it doesn't work out of the box, meaning having to apply downstream patches/workarounds. You could start to provide a release tarball containing everything required (source & pre-generated files). The default for release(d) versions should be to pick up the pre-generated files automatically if available and for scm/development builds rebuilding them if there are no pre-generated files available or maybe add a CMake option to force rebuilding them for better control.

[hsntgm]

@bjoernricks it seems you have updated node modules, e.g. core-js is now at version 3.2.1 which was 2.6.5 for old gsa 8.0.0.

I didn't check all of them but i get lots of missing node modules errors while compiling gsa 9.0.0 wtih previous node-modules that you uploaded.
(https://github.com/greenbone/gsa/releases/download/v8.0.1/gsa-node-modules-8.0.1.tar.gz).

Failed to compile.
./src/gmp/log.js
Cannot find module: 'core-js/modules/es.object.entries'. Make sure this package is installed.
You can install this package by running: yarn add core-js/modules/es.object.entries.

Could you provide node-modules tarball for 9.0.0 ?
~Hasan

[bjoernricks]

@hsntgm https://github.com/greenbone/gsa/releases/download/v9.0.0/gsa-node-modules-9.0.0.tar.gz

[hsntgm]

@bjoernricks
Thanks.