Upgrading the SK server using provided functions impossible

[ghost]

On the server, the GUI suggested a update from 1.34.0 to 1.35.2. Using GUI failed by getting stuck without saying anything after running for a while in the CLI. After which, the 'signalk-server' became unusable, impossible to restart.

Reinstall all from the .exe seems to be the only option to get updates (or from the command line but that breaks the idea of the usability of this package, I reckon).

For the reinstall, also an issue for the end user was then detected, on

https://github.com/SignalK/signalk-server-windows/releases/tag/v0.3.2

on October version Win10 20H2 MS Defender finds Trojan:Win32/Zpevdo.B from the installer and removes the download at early stage.

I am not using this Signal K for navigation (yes, I have a RPI) but for SW testing. I have the means to fix the issue myself. However, this is likely to happen as well to a non-developer who would give up, perhaps?

[ghost]

Manual reparation as below, using truncated PATH not containing my settings:

c:\signalk\signalkhome>echo %NODE_PATH%
c:\signalk\nodejs
c:\signalk\signalkhome>set PATH=%NODE_PATH%;c:\signalk\openssl\bin
C:\signalk\tools\npm-install-signalk-server.cmd

Which runs fine except with some error messages because the truncated PATH excludes all my own stuff, including Python:

...
c:\signalk\nodejs\node_modules\signalk-server\node_modules\mdns>if not defined npm_config_node_gyp (node "c:\signalk\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "c:\signalk\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
gyp ERR! find Python
gyp ERR! find Python Python is not set from command line or npm configuration
gyp ERR! find Python Python is not set from environment variable PYTHON
gyp ERR! find Python checking if "python" can be used
gyp ERR! find Python - "python" is not in PATH or produced an error
....
+ signalk-server@1.35.2
added 493 packages from 330 contributors and updated 1 package in 110.054s

Now, execution:

c:\signalk\nodejs>signalk-server.cmd
signalk-server running at 0.0.0.0:3000

ENOENT: no such file or directory, open 'c:\signalk\nodejs\node_modules\signalk-server\node_modules\@signalk\aava-n2k.data'
...
Otherwise OK, but nicely the reinstall cleans @signalk in node_modules, in which I kept my data files...

[cmotelet]

Hi @Canne,
thanks for this issue.
For the virus problem, it is a false positive.
I got it at the beginning with Avast, I sent the file to support for an in-depth analysis and since then it seems that Avast does not report it anymore (I use Avast Premium Secury).
Obviously for Microsoft Defender, it's more complicated.
To submit a file, I need an active Software Assurance that I don't have.
When I scan it manually with Defender, it doesn't alert me. But since Defender is the second anti-virus on the download, it does not scan the file.
I'm still looking for a solution to this problem but I haven't found a magic solution yet. Anti-virus companies don't like NSIS very much.
I will try to deploy an EC2 AWS instance with a W2K16, I think Defender is included in it, and do some more tests...

[cmotelet]

For the broken update problem, I already had this case and it seems to be caused by the use of the example file by SignalK (aava-n2k.data) during the update.
NPM uninstalls SignalK but falls into error, because it can't delete the file with a lock and then the update stops.
Then it is impossible to restart because NPM uninstalled SignalK but did not reinstall it.
Can you give me your settings.json file for analyse ? (Here or on SLACK)
I'm going to try to reproduce precisely this problem and if it's the case, see with @tkurki if we can act on the server code to avoid it.

[cmotelet]

And finally, command line PATH concatenation is indeed a problem when the PATH is very long (2047).
https://docs.microsoft.com/en-us/troubleshoot/windows-client/shell-experience/command-line-string-limitation

Can you give me your complete PATH before transformation so that I can do some tests? (Here or on SLACK)

[ghost]

Hi, the %PATH% is in Slack DM now. If that is the root cause, maybe the following, in your script need to be changed:

set PATH=%NODE_PATH%;c:\signalk\openssl\bin;%~dp0;%PATH%

to

set PATH=%NODE_PATH%;c:\signalk\openssl\bin;%~dp0

and then make evolve the PATH so that it is self contained, making that window to run totally out of the user (messy) context, i.e. having no dependencies on third party programs.

[ghost]

BTW, the outcome of your work (and others) is so stable and soooo useful: since I am in USB and network streaming only, I think I will get rid off the RPI on my boat and run this one instead on my navigation Win 10 station which is turned on all the time, anyway. Absolutely great job, especially that "run as service" - job you did. Thanks a bunch!

[cmotelet]

@Canne ,
on upgrade failure, I've opened an issue SignalK/signalk-server#1170.
We will try to see if we can fix in the server code.

[cmotelet]

on Microsoft Defender problem, I can now reproduce it in my virtual machine and submit a request to the Microsoft team.
I'm building a new release with some fixes trying to go under the radar.
Then I will submit the new release to them.
I hope they will hear me

[ghost]

Great, thanks! I will, of course help in testing later on. Right now I have signalk-server 1.35.2.

[cmotelet]

@Canne ,
about upgrade failure, it will be fixed in the next signalk-server version 1.36.0 (SignalK/signalk-server#1172).
But beware, when you upgrade from version 1.35.2 to version 1.36.0, remember to disable your FileStream connections in Data Connections.
After version 1.36.0 this will no longer be necessary.

[cmotelet]

Regarding virus detection in the installer by Microsoft Defender, I have just released version 0.4.0.
https://github.com/SignalK/signalk-server-windows/releases/tag/v0.4.0
It seems to generate fewer false positives:

And now Microsoft Defender report only:

I have to report to the Microsoft team to try to get on their white list...

[ghost]

Great! I will observe the SIgnal K node server notifies me about the update being available and will test as you suggested. Will report back here, of course. Do you want me first to upgrade or even uninstall the current version and reinstall (wipe out the old) with 0.4.0?

It actually downloads w/ Vivaldi (=Chrome) no pb. A scan with my (earlier picky Defender) gives now:

[cmotelet]

As nodeJs and Openssl have been updated with the latest version, I think it's better to delete everything.
Save before the contents of C:\signalk\signalkhome\.signalk
Stop the SignaK service: C:\signalk\tools\stop-signalk-server-services.cmd
Remove the SignalK service: C:\signalk\tools\remove-signalk-server-services.cmd
Re-install with all options
Then restore your config backup C:\signalk\signalkhome\.signalk
If you want, you can also try the ssl option in Server/Settings/Server Settings

[cmotelet]

I just thought SignalK now has a function to save the configuration in Server/Backup/Restore.
It can be useful to try it also.

[ghost]

Hi, after theServer/Backup removed the entire folder signalk as suggested in C:\signalk\readme.html (better put that in the on-line help since reading the file locks the folder...).

Downloading (as above) 0.4.0 no messages from Defender, just the usual warning from Vivaldi as for all *.exe.

Installation is faster and smaller in size 256MB than before (was +500MB), # of files 30k vs +50k before.

Restoration of settings (boat) and plug-ins (could not get plug-in list though in server settings but after a restart) worked, and the admin user was there. However, connection settings and my test data recording files were not there after the Restore and I restored those manually from my own backups. I am not sure in which path they supposed to be, in C:\signalk\signalkhome to get backed up?

SSL works on localhost:3443 OK.

I am now on 1.35.2 - let's wait for the server upgrade being suggested to resolve the original issue of upgrade freezing.

Meanwhile, thanks a lot for your great support and effort!

[cmotelet]

The C:\signalk\readme.html file is a transformation in HTML of the readme.md file present on the git.
I wanted to integrate it in the kit to have instructions in offline mode. But it's true that it can be surprising when you try to erase if this file is open.
If only this one and the folder are left, it's not too serious.

[cmotelet]

For the backup, it only takes the json files and plugins present in C:\signalk\signalkhome\.signalk\ so it is logical that your file was not caught by the backup.

[cmotelet]

If you don't want to be impacted anymore by the bug of the files open at the next update, you can manually switch to the beta version which already has the patch.

• stop-signalk-server-services.cmd
• Open a SignalK CLI
• npm install -g --unsafe-perm signalk-server@1.36.0-beta.3
• On next release (1.36.0), you will leave automaticaly the beta branch and return in normal branch.

If you want revert to the actual version: npm install -g --unsafe-perm signalk-server

[ghost]

Hi, I suggest that you resolve this thread which has grown out of its title:

I am pleased to confirm that I have successfully finished all the tests of my own software using your very stable and easy to use packaging for Windows in the execution of my test plans needing Signal K data. Thank you so much!

I wanted to finish by contributing to your work and to this ticket but unfortunately the update from 1.35.2 to 1.37.2 failed and for the exactly same reason and exactly the same way - I have been looping aavan2k.data the same.

But that is not important if the update works otherwise, my use case is not the usual one, of course.

Finally and FYI, in my own software NSIS-built *.exe installer I encounter similar warnings from Defender whenever I have node_modules in it. I took them away, the Defender is happy. They just don't like some pattern and/or scripting in general and declare some creepy Trojan horse. So, also from that point of view, if you like you can resolve this ticket.

Once again, many thanks for this very useful work!