Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

source a .conf file with skywire-cli config gen #1558

Merged
merged 39 commits into from
Jul 7, 2023
Merged

source a .conf file with skywire-cli config gen #1558

merged 39 commits into from
Jul 7, 2023

Conversation

0pcom
Copy link
Collaborator

@0pcom 0pcom commented May 15, 2023

skywire-cli config gen has a lot of flags, but still lacks the granularity of configuration which may be desired to persist or set initially for config generation.

This PR introduces a comment-able, human-editable standard .conf template which can be specified as an environmental variable to config gen and sourced (or the windows / powershell equivalent)

Here is the WIP version for linux

$ go run cmd/*/*cli.go config gen -q

#
# /etc/skywire.conf
#
#########################################################################
#	SKYWIRE CONFIG TEMPLATE
#		Defaults for booleans are false
#		Uncomment to change default value
#########################################################################

#--	Other Visors will automatically establish transports to this visor
#	requires port forwarding or public ip
#VISORISPUBLIC=true

#--	Autostart vpn server for this visor
#VPNSERVER=true

#--	Use test deployment
#TESTENV=true

#--	Automatically determine the best protocol (dmsg or http)
#	based on location to connect to the deployment servers
#BESTPROTO=true

#--	Set custom service conf URLs
#SVCCONFADDR=('')

#--	Set visor runtime log level.
#	Default is info ; uncomment for debug logging
#LOGLVL=debug

#--	Use dmsghttp to connect to the production deployment
#DMSGHTTP=true

#--	Start the hypervisor interface for this visor
#ISHYPERVISOR=true

#--	Output path of the config file
#OUTPUT='./skywire-config.json'

#--	Display the node ip in the service discovery
#	for any public services this visor is running
#DISPLAYNODEIP=true

#--	Set remote hypervisor public keys
#HYPERVISORPKS=('')

#--	Default config paths for the installer or package (system paths)
#PKGENV=true

#--	Default config paths for the current userspace
#USRENV=true

#--	Set secret key
#SK=''

#--	Disable auto-transports to public visors
#DISABLEPUBLICAUTOCONN=true

#--	Custom config version override
#VERSION=''

#--	Set app bin_path
#BINPATH='./apps'

This will be extended to allow for or include

Invocation

to generate a json skywire config using such a template, the path to the .conf file should be specified as an env to skywire-cli config gen as follows

SKYENV=/path/to/skywire.conf skywire-cli config gen

The values override the defaults of the flags, viewable with config gen --all as demonstrated in the following screenshots:
no env file specified or file not detected
image

Here I have

  • set a hypervisor public key
  • set a service conf url
  • changed the log level
  • disabled public autoconnect

image

Usage

The use of the env file and flags will be to a large extent mutually exclusive with flags, as using the same flagset (for boolean values) as envs specified in the .config will cancel each other out. Some flags will not be made available to set in the env file; specifically the following flags

  • -n stdout
  • -x retain hypervisors
  • -r regenerate config or retain keys

additionally, more flags need to be added to achieve the desired level of granularity of configuration.

Non-linux implementations

  • The dot (.) command on windows (powershell) is the equivalent of the source command on linux.
  • The envfile format needs to be altered to have $ in front of variable declarations
  • commenting otherwise remains the same

@0pcom 0pcom changed the title source a .conf file with skywire-cli config gen [WIP] source a .conf file with skywire-cli config gen May 23, 2023
@0pcom
Copy link
Collaborator Author

0pcom commented May 23, 2023

image

Generate a config file

	Config defaults file may also be specified with
	SKYENV=/path/to/skywire.conf skywire-cli config gen

Usage:
  cli config gen [flags] 

Flags:
  -a, --url string           services conf url
 (default "http://conf.skywire.skycoin.com")
      --loglvl string        level of logging in config (default "info")
  -b, --bestproto            best protocol (dmsg | direct) based on location
  -c, --noauth               disable authentication for hypervisor UI
  -d, --dmsghttp             use dmsg connection to skywire services
  -e, --auth                 enable auth on hypervisor UI
  -f, --force                remove pre-existing config
  -g, --disableapps string   comma separated list of apps to disable
  -i, --ishv                 local hypervisor configuration
  -j, --hvpks string         list of public keys to add as hypervisor
      --dmsgpty string       add dmsgpty whitelist PKs
      --survey string        add survey whitelist PKs
      --routesetup string    add route setup node PKs
      --tpsetup string       add transport setup node PKs
  -k, --os string            (linux / mac / win) paths (default "linux")
  -l, --publicip             allow display node ip in services
  -m, --example-apps         add example apps to the config
  -n, --stdout               write config to stdout
  -o, --out string           output config: skywire-config.json
  -p, --pkg                  use path for package: /opt/skywire
  -u, --user                 use paths for user space: /root
  -r, --regen                re-generate existing config & retain keys
  -s, --sk cipher.SecKey     a random key is generated if unspecified
 (default 0000000000000000000000000000000000000000000000000000000000000000)
  -t, --testenv              use test deployment conf.skywire.dev
  -v, --servevpn             enable vpn server
  -w, --hide                 dont print the config to the terminal :: show errors with -n flag
  -x, --retainhv             retain existing hypervisors with regen
  -y, --autoconn             disable autoconnect to public visors
  -z, --public               publicize visor in service discovery
      --stcpr int            set tcp transport listening port - 0 for random
      --sudph int            set udp transport listening port - 0 for random
      --binpath string       set bin_path
  -q, --envs                 show the environmental variable settings
      --nofetch              do not fetch the services from the service conf url
      --nodefaults           do not use hardcoded defaults for production / test services
      --version string       custom version testing override

Copy link
Member

@jdknives jdknives left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will have to do a review in multiple parts.

pkg/visor/visorconfig/v1.go Outdated Show resolved Hide resolved
pkg/visor/visorconfig/v1.go Outdated Show resolved Hide resolved
func MustPKs(pks string) []cipher.PubKey {
var sPKs cipher.PubKeys
if err := sPKs.Set(pks); err != nil {
fmt.Printf("invalid public key or keys: %s", pks)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that this package uses fmt consistently. Is there a reason we are using fmt here instead of the logging pacakge we use typically?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no logging package already used in this particular file, but fmt was already imported

i was replacing a panic with an actual error before the panic because at one point i was hitting that panic and it was very unhelpful.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@0pcom we use the logging package liberally throughout the project. I dont see a reason we should avoid importing it here unless you see an explicit reason.

panic(err)
}

return sPK
}

// MustPKs unmarshals comma separated list of string PKs to []cipher.PubKey. It panics if unmarshaling fails.
func MustPKs(pks string) []cipher.PubKey {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we name this more explicitly like MustParsePKs?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to rename the one above it to match as MustParsePK

cmd/skywire-cli/commands/config/gen.go Show resolved Hide resolved
cmd/skywire-cli/commands/config/gen.go Outdated Show resolved Hide resolved
cmd/skywire-cli/commands/config/gen.go Outdated Show resolved Hide resolved
cmd/skywire-cli/commands/config/gen.go Outdated Show resolved Hide resolved
if err != nil {
log.WithError(err).Fatal("Failed to create http request\n")
}
req.Header.Add("Cache-Control", "no-cache")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is the cache supposed to come from? I dont think we are currently caching the conf service responses.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot say, this was existing code I just moved

cmd/skywire-cli/commands/config/gen.go Outdated Show resolved Hide resolved
Copy link
Member

@jdknives jdknives left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will have to do a review in multiple parts.

@jdknives jdknives deleted the branch skycoin:develop June 1, 2023 13:03
@jdknives jdknives closed this Jun 1, 2023
@0pcom 0pcom reopened this Jun 3, 2023
cmd/skywire-cli/commands/config/gen.go Outdated Show resolved Hide resolved
gHiddenFlags = append(gHiddenFlags, "binpath")
// genConfigCmd.Flags().StringVar(&addSkysocksClientSrv, "proxyclientpk", scriptExecString("${PROXYCLIENTPK}"), "set server public key for proxy client")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still dead code here.

if err != nil {
//silence errors for stdout
if !isStdout {
log.WithError(err).Error("Failed to fetch servers\n")
log.Warn("Falling back on hardcoded servers")
}
} else {
// nil error from client.Do(req)
// nil error from client.Get
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move all of this code to inside the else clause outside the else clause? Resp body should be closed even if there is an error I believe. Also, can we set the log level to panic or something equivalent and remove all these checks for !stdout?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

valid points, will make the changes

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've outdented the code as requested, but in terms of getting rid of 'all these checks for !isStdout' it's not just as straightforward as changing the log level because the behavior is also changed by the stdout flag to not create or read files and I already have the -w flag as an override for showing the errors, so it's something I would like to revisit after this PR is merged.

We are talking about code that currently works and is on develop in any case

@0pcom
Copy link
Collaborator Author

0pcom commented Jun 28, 2023

image

I've gone ahead and implemented the dead code which was functionality from config update

also updated the env file

#
# /etc/skywire.conf
#
#########################################################################
#	SKYWIRE CONFIG TEMPLATE
#		Defaults for booleans are false
#		Uncomment to change default value
#########################################################################

#--	Other Visors will automatically establish transports to this visor
#	requires port forwarding or public ip
#VISORISPUBLIC=true

#--	Autostart vpn server for this visor
#VPNSERVER=true

#--	Use test deployment
#TESTENV=true

#--	Automatically determine the best protocol (dmsg or http)
#	based on location to connect to the deployment servers
#BESTPROTO=true

#--	Set custom service conf URLs
#SVCCONFADDR=('')

#--	Set visor runtime log level.
#	Default is info ; uncomment for debug logging
#LOGLVL=debug

#--	Use dmsghttp to connect to the production deployment
#DMSGHTTP=true

#--	Start the hypervisor interface for this visor
#ISHYPERVISOR=true

#--	Output path of the config file
#OUTPUT='./skywire-config.json'

#--	Display the node ip in the service discovery
#	for any public services this visor is running
#DISPLAYNODEIP=true

#--	Set remote hypervisor public keys
#HYPERVISORPKS=('')

#--	Default config paths for the installer or package (system paths)
#PKGENV=true

#--	Default config paths for the current userspace
#USRENV=true

#--	Set secret key
#SK=''

#--	Disable auto-transports to public visors
#DISABLEPUBLICAUTOCONN=true

#--	Custom config version override
#VERSION=''

#--	Set app bin_path
#BINPATH='./apps'

#--	Set server public key for proxy client to connect to
#PROXYCLIENTPK=''

#--	Enable autostart of the proxy client
#STARTPROXYCLIENT=true

#--	Disable autostart of proxy server
#NOPROXYSERVER=true

#--	Set a password for the proxy server
#PROXYSEVERPASS=''

#--	Password for the proxy client to access the server (if password is set for the server)
#PROXYCLIENTPASS=''

#--	Set VPN client killswitch
#VPNKS=true

#--	Set vpn server public key for the vpn client to use
#ADDVPNPK=''

#--	Password for vpn client to access the server (if password is set forthe server)
#VPNCLIENTPASS=''

#--	Set password to the vpn server
#VPNSEVERPASS=''

#--	Change secure mode status of vpn server
#VPNSEVERSECURE=''

#--	Set VPN Server network interface
#VPNSEVERNETIFC=''

@0pcom 0pcom merged commit c36d0b3 into skycoin:develop Jul 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants