Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[jcreate] `extract` option cannot be used with `inter=0` #367

Closed
am11 opened this issue Jan 1, 2019 · 49 comments
Assignees

Comments

@am11
Copy link

@am11 am11 commented Jan 1, 2019

Background

FreeBSD version (uname -a):

FreeBSD bazinga.localdomain 11.2-RELEASE FreeBSD 11.2-RELEASE #0 r335510: Fri Jun 22 04:32:14 UTC 2018 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64

CBSD version (cbsd version):

12.0.3

Firstly, thank you for this tool! Its ability to run in non-interactive shell, download and use distribution release binaries feature in jcreate, are some rare gems! 💎

On a fresh FreeBSD amd64 system, a script takes precisely seven steps to get into ia32 jail:

Code (click or tap to expand)
#!/usr/bin/env sh

# 0. prepare skel directory '/tmp/s/g/x' (git clone etc.)

# 1. install package
sudo pkg install -y cbsd

# 2. initialize with a random(?) workdir
sudo workdir=/tmp /usr/local/cbsd/sudoexec/initenv /usr/local/cbsd/share/initenv.conf

# 3. create a jconf
# (copied from Oleg's example in gh#34,
# can we have this i386 conf in samples, so we can just copy it and override
# its values during jcreate?)
cat > /tmp/jail-11i386.jconf << EOF
jname="jail-11i386";
path="/usr/jails/jails/jail-11i386";
host_hostname="jail-11i386.my.domain";
ip4_addr="10.0.0.4/16";
mount_devfs="1";
allow_mount="1";
allow_devfs="1";
allow_nullfs="1";
mount_fstab="/usr/jails/jails-fstab/fstab.jail-11i386";
arch="i386";
mkhostsfile="1";
devfs_ruleset="4";
ver="11.2";
basename="";
baserw="0";
mount_src="0";
mount_obj="0";
mount_kernel="0";
mount_ports="1";
astart="1";
data="/usr/jails/jails-data/jail-11i386-data";
vnet="0";
applytpl="1";
mdsize="0";
rcconf="/usr/jails/jails-rcconf/rc.conf_jail-11i386";
floatresolv="1";

exec_poststart="0";
exec_poststop="";
exec_prestart="0";
exec_prestop="0";

exec_master_poststart="0";
exec_master_poststop="0";
exec_master_prestart="0";
exec_master_prestop="0";
pkg_bootstrap="1";
user_pw_root='rootpw'
interface="auto"
jailskeldir="/tmp/s/g/x"
pkglist="/tmp/pkglist.txt";
exec_start="/bin/sh /etc/rc"
exec_stop="/bin/sh /etc/rc.shutdown"
EOF

# 4. specify required packages
cat > /tmp/pkglist.txt << EOF
c-ares
gmake
icu
libnghttp2
libuv
git
EOF

# 5. create a jail (not the use of `inter=0` for non-interactive scenario)
sudo cbsd jcreate jconf=/tmp/jail-11i386.jconf inter=0 arch=i386

# 6. start the jail
sudo cbsd jstart jail-11i386

# 7. enter jail-i386 for further execution
sudo cbsd jexec jname=jail-11i386 /bin/tcsh

There is an explicit check in system.subr for inter=0 (step 5 above), that it has to use repo option to get the system. By reading the code further in :/tools/repo file, it seems like the repo option causes CBSD to download binaries from the internet.

Feature Request

A way for non-interactive shell to re-use the pre-downloaded files, without downloading from the internet again.

Something like:

sudo cbsd jcreate jconf=/tmp/jail-11i386.jconf inter=0 arch=i386 archivedir=/tmp/archives/fbsd-i386/

and/or better yet, if we already have it extracted as well, could we use it? e.g. in /tmp/fbsd-i386/ directory, user could untar the base.txz and use it multiple times in jcreate:

sudo cbsd jcreate jconf=/tmp/jail-11i386.jconf inter=0 arch=i386 systemdir=/tmp/fbsd-i386/
@olevole olevole self-assigned this Jan 2, 2019
@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 2, 2019

Hi Adeel! Thanks for ideas/request. Yes, it seems reasonable.

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 2, 2019

notes for myself: CBSD can use multiple sources to receive bases ( from archive, from network, from current system ..).. maybe we can do config file to control the default action in non-interactive mode with archivedir/cachedir optional settings ?

olevole added a commit that referenced this issue Jan 5, 2019
…e_source_by_list

start working on Issue #367:
lets do the default behavior more flexible and customizable by gloabal config: bases.conf
todo: maybe it makes sense to add per-jail overwrites ?
todo: support for $ver/$arch variables in config for per-ver, per-arch sources
todo: support for list of sources

Work-In-Progress
Suggested by: Adeel Mujahid (@am11)
olevole added a commit that referenced this issue Jan 7, 2019
- switch choose method to select_jail tools (todo: rename it to something neutral)
- config for bases per-platform, e.g: FreeBSD-bases.conf, TrueOS-bases.conf, HardenedBSD-bases.conf
- default_obtain_base_method can be list (for non-interactive method). In interactive mode always
    prefer for first
- select_jail.c: add default options for selector via argv[4]

WIP (work-in-prgogress), Issue #367
olevole added a commit that referenced this issue Jan 9, 2019
   - update exract mode method to support multiple sources

todo: update repo for custom url= args to not break the old behavior

WIP, Issue #367
olevole added a commit that referenced this issue Jan 9, 2019
- repo: add optional url= args for custom url

WIP, Issue #367
@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 9, 2019

it seems it should work now, work is almost complete.

what we have now:

  1. old behavior of repo script remained (except new args= optional arg) untouched (so far) so as not to violate the old behavior.

  2. we have per-platform config files where CBSD global variables like ${platform}, ${ver}, ${arch} and ${target_arch} should work.

what we lost:

New way does not take into account that may exist lib32.txz part of base. I think this can be neglected ( who want lib32.txz - can comment out default_obtain_base_repo_sources= variable in config file to restore original behavior where 'repo' script tries to get lib32.txz...

  1. For difference version URL may differ. For example, base.txz for RELEASES placed on

/releases/${arch}/${target_arch}/${ver}-RELEASE, e.g:

https://download.freebsd.org/ftp/releases/arm64/aarch64/12.0-RELEASE/
https://download.freebsd.org/ftp/releases/amd64/amd64/12.0-RELEASE/

while CURRENT places here:

/releases/${arch}/${target_arch}/${ver}-RELEASE, e.g:

/snapshots/${arch}/${target_arch}/${ver}-CURRENT/

e.g:

https://download.freebsd.org/ftp/snapshots/amd64/amd64/13.0-CURRENT/

Due to CBSD use '.' to include config files, we can repeat 'the magic' from 'repo' script with sh-based condition/switch:

case ${ver} in
   13)
         ../snapshot/
    *)
        ../release/

See: https://github.com/cbsd/cbsd/blob/develop/etc/defaults/FreeBSD-bases.conf

this is cool, but for users who expect a static configuration file, this may be unusual. So what to do ;-)

In any case, the new scheme is more flexible.

@mekanix I do not have any HardenedBSD platform. Can you test latest code on HBSD ? Just remove old base via

'cbsd removebase'
'cbsd removebase ver=XX stable=YY'

and try to create jail with 11.2 and 12.0 version then start jail.

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 9, 2019

@am11 If you also want to test this code, you can try switch to GIT develop version:


pkg remove -f cbsd
rm -rf /usr/local/cbsd
git clone https://github.com/cbsd/cbsd.git /usr/local/cbsd

# restore rc.d and bsdconfig symlink:
cd /usr/local/etc/rc.d
ln -sf /usr/local/cbsd/rc.d/cbsdd
mkdir -p /usr/local/libexec/bsdconfig
cd /usr/local/libexec/bsdconfig
ln -s /usr/local/cbsd/share/bsdconfig/cbsd

And re-run 'cbsd initenv':

/usr/local/cbsd/sudoexec/initenv

old environment/workdir should catch up without any loss ( you can not even stop running containers (I think)). But the backup doesn't hurt of course.

After upgrade, you can customize FreeBSD-bases.conf as you wish:

cp ~cbsd/etc/defaults/FreeBSD-bases.conf ~cbsd/etc/
vi ~cbsd/etc/FreeBSD-bases.conf

(edit in ~cbsd/etc/FreeBSD-bases.conf, not ~cbsd/etc/defaults/FreeBSD-bases.conf)

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 9, 2019

my personal todo: add support for multiple method

https://github.com/cbsd/cbsd/blob/develop/etc/defaults/FreeBSD-bases.conf#L7

e.g can be:

default_obtain_base_method="extract repo"
default_obtain_base_repo_sources=/usr/freebsd-dist/base.txz"
default_obtain_base_extract_source="http://my-private-server/$platform-base.txz"

And non-interactive mode try to first get:

/usr/freebsd-dist/base.txz and when no file, try to get from http://my-private-server/FreeBSD-base.txz

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 9, 2019

it also seems to make sense to add a check that version (and arch) in base.txz is equal expected ${ver} and ${arch}. it's easy to do through version from ELF header in /bin/sh from base.txz:

~cbsd/misc/elf_tables --ver /bin/sh
~cbsd/misc/elf_tables --freebsdver /bin/sh
~cbsd/misc/elf_tables --arch /bin/sh

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 9, 2019

comments are welcome

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 12, 2019

close this issue, will be included in 12.0.4. If you find problems - please reopen, thanks.

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 27, 2019

@olevole, thanks a lot! I was busy with other projects and only today had a time to get back to FreeBSD and this issue.

I have tested it in a fresh FreeBSD 11 amd64 using the following steps:

# entered FreeBSD 11 first time after the install

# install packages
sudo pkg update
sudo pkg install -y sudo libssh2 rsync sqlite3 git pkgconf

# clone and setup cbsd
sudo git clone https://github.com/cbsd/cbsd.git /usr/local/cbsd --single-branch --branch master --depth 1
cd /usr/local/etc/rc.d
sudo ln -sf /usr/local/cbsd/rc.d/cbsdd
sudo mkdir -p /usr/local/libexec/bsdconfig
cd /usr/local/libexec/bsdconfig
sudo ln -s /usr/local/cbsd/share/bsdconfig/cbsd
sudo pw useradd cbsd -s /bin/sh -d /nonexistent -c "cbsd user"

# initialize
sudo workdir=/tmp /usr/local/cbsd/sudoexec/initenv /usr/local/cbsd/share/initenv.conf

# create base config
sudo bash -c 'cat > /usr/local/cbsd/etc/FreeBSD-bases.conf' << EOF
default_obtain_base_method="extract repo"
default_obtain_base_repo_sources="/usr/freebsd-dist/base.txz"
default_obtain_base_extract_source="https://github.com/am11/freebsd-dist/releases/download/11.2/base-i386.txz"
EOF

# create jail config
cat > /tmp/jail-11i386.jconf << EOF
jname="jail-11i386";
path="/usr/jails/jails/jail-11i386";
host_hostname="jail-11i386.my.domain";
ip4_addr="10.0.0.4/16";
mount_devfs="1";
allow_mount="1";
allow_devfs="1";
allow_nullfs="1";
mount_fstab="/usr/jails/jails-fstab/fstab.jail-11i386";
arch="i386";
mkhostsfile="1";
devfs_ruleset="4";
ver="11.2";
basename="";
baserw="0";
mount_src="0";
mount_obj="0";
mount_kernel="0";
mount_ports="1";
astart="1";
data="/usr/jails/jails-data/jail-11i386-data";
vnet="0";
applytpl="1";
mdsize="0";
rcconf="/usr/jails/jails-rcconf/rc.conf_jail-11i386";
floatresolv="1";

exec_poststart="0";
exec_poststop="";
exec_prestart="0";
exec_prestop="0";

exec_master_poststart="0";
exec_master_poststop="0";
exec_master_prestart="0";
exec_master_prestop="0";
pkg_bootstrap="1";
user_pw_root='rootpw'
interface="auto"
jailskeldir="/tmp/s/g/x"
pkglist="/tmp/pkglist.txt";
exec_start="/bin/sh /etc/rc"
exec_stop="/bin/sh /etc/rc.shutdown"
EOF

sudo cbsd jcreate jconf=/tmp/jail-11i386.jconf inter=0 arch=i386

the output suggests that it is not using the values from /usr/local/cbsd/etc/FreeBSD-bases.conf (github releases URL):

[am11@vdnframe5 /usr/local/libexec/bsdconfig]$ sudo cbsd jcreate jconf=/tmp/jail-11i386.jconf inter=0 arch=i386
firstboot-growfs=YES: not found
No base dir in: /usr/jails/basejail/base_i386_i386_11.2
config-based sources: https://download.freebsd.org/ftp/releases/i386/i386/11.2-RELEASE/base.txz http://ftp.freebsd.org/pub/FreeBSD/releases/i386/i386/11.2-RELEASE/base.txz https://pub.allbsd.org/pub/FreeBSD/releases/i386/i386/11.2-RELEASE/base.txz
No such bases here: /usr/jails/basejail/base_i386_i386_11.2
Scanning for fastest mirror...
 * [ 1/3   ] https://download.freebsd.org/ftp/releases/i386/i386/11.2-RELEASE/base.txz:   454656
 * [ 2/3   ] http://ftp.freebsd.org/pub/FreeBSD/releases/i386/i386/11.2-RELEASE/base.txz: 692224
 * [ 3/3   ] https://pub.allbsd.org/pub/FreeBSD/releases/i386/i386/11.2-RELEASE/base.txz: 16384
 Winner: http://ftp.freebsd.org/pub/FreeBSD/releases/i386/i386/11.2-RELEASE/base.txz
Looking for official FreeBSD mirror:
Retrieve base.txz from ftp.freebsd.org, size: 85m
/usr/jails/tmp/src.5204/base.txz                1% of   84 MB  363 kBps 03m28s

Same result if I add:

default_obtain_base_method="extract repo"
default_obtain_base_repo_sources="/usr/freebsd-dist/base.txz"
default_obtain_base_extract_source="https://github.com/am11/freebsd-dist/releases/download/11.2/base-i386.txz"

in /tmp/jail-11i386.jconf.

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 27, 2019

Hmm:

a) sudo workdir=/tmp /usr/local/cbsd/sudoexec/initenv /usr/local/cbsd/share/initenv.conf

b) 'cat > /usr/local/cbsd/etc/FreeBSD-bases.conf' << EOF
...

you must write custom configurations in $workdir/etc, not /usr/local/cbsd
In your case, /tmp/etc/defaults/ - default config file for environment which can be overridden in /tmp/etc/ dir.

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 27, 2019

try:
sudo workdir=/tmp /usr/local/cbsd/sudoexec/initenv /usr/local/cbsd/share/initenv.conf

b) cat > /tmp/etc/FreeBSD-bases.conf << EOF
default_obtain_base_method="extract repo"
default_obtain_base_repo_sources="/usr/freebsd-dist/base.txz"
default_obtain_base_extract_source="https://github.com/am11/freebsd-dist/releases/download/11.2/base-i386.txz"
EOF

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 27, 2019

Thanks. I recreated a new machine this time with symlink /tmp/etc -> /usr/local/cbsd/etc, and # create base config step before # initialize; it is now picking up github.com, however hanged indefinitely:

[am11@vdnframe5 /tmp]$ sudo cbsd jcreate jconf=/tmp/jail-11i386.jconf inter=0 arch=i386
firstboot-growfs=YES: not found
No base dir in: /usr/jails/basejail/base_i386_i386_11.2
Scan for config-based path to base archive: https://github.com/am11/freebsd-dist/releases/download/11.2/base-i386.txz
info: no such archive file: https://github.com/am11/freebsd-dist/releases/download/11.2/base-i386.txz
Please provide full path to base archive, (e.g. default: /usr/freebsd-dist/base.txz):

same thing with bintray URL (which is much faster than github/aws, which in turn is much faster than all ftp.freebsd mirrors), if i download the archive manually before # initialize:

sudo fetch https://bintray.com/am11/freebsd-dist/download_file?file_path=base-11.2-i386.txz -o /usr/freebsd-dist/base.txz

it still hits the URL although /usr/freebsd-dist/base.txz is a valid archive. Please find the attachment for updated sh.

fbsd-jailsetup.zip

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 27, 2019

I also realized that fetchme function is using fetch -qs to get size, however, github releases link returns forbidden for resource size quires:

[am11@vdnframe5 /tmp]$ fetch -vs https://github.com/am11/freebsd-dist/releases/download/11.2/base-i386.txz
resolving server address: github.com:443
SSL options: 83004bff
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES128-GCM-SHA256
Certificate subject: /businessCategory=Private Organization/jurisdictionC=US/jurisdictionST=Delaware/serialNumber=5157550/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
Certificate issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Extended Validation Server CA
requesting https://github.com/am11/freebsd-dist/releases/download/11.2/base-i386.txz
302 redirect to https://github-production-release-asset-2e65be.s3.amazonaws.com/164260188/fc5a0e80-115a-11e9-9b4d-2577c642a25a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190127T120736Z&X-Amz-Expires=300&X-Amz-Signature=bb84437e3d2eb051c16bb921d34f833538e48a956055f7c4ed48c442d93a2c8d&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dbase-i386.txz&response-content-type=application%2Foctet-stream
resolving server address: github-production-release-asset-2e65be.s3.amazonaws.com:443
SSL options: 83004bff
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES128-GCM-SHA256
Certificate subject: /C=US/ST=Washington/L=Seattle/O=Amazon.com Inc./CN=*.s3.amazonaws.com
Certificate issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Baltimore CA-2 G2
requesting https://github-production-release-asset-2e65be.s3.amazonaws.com/164260188/fc5a0e80-115a-11e9-9b4d-2577c642a25a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190127T120736Z&X-Amz-Expires=300&X-Amz-Signature=bb84437e3d2eb051c16bb921d34f833538e48a956055f7c4ed48c442d93a2c8d&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dbase-i386.txz&response-content-type=application%2Foctet-stream
fetch: Forbidden

Edit: normal download (without -s) works fine:
fetch -v https://github.com/am11/freebsd-dist/releases/download/11.2/base-i386.txz

olevole added a commit that referenced this issue Jan 27, 2019
also for inter=0 we go through all the values in default_obtain_base_method vars
@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 27, 2019

@am11 I fixed a few problems related to this issue. Can you 'git pull' + 'cbsd initenv' and try again ?
Also i found in your config mistake:

default_obtain_base_extract_source= must point to a local path (extract source)
default_obtain_base_repo_sources= must point to a remote url (repo source)

your option is opposite ;)

Relevant config file in my latest test:

cat ~cbsd/etc/FreeBSD-bases.conf

auto_baseupdate=0
default_obtain_base_method="extract repo"
default_obtain_base_extract_source="/usr/freebsd-dist/base.txz"
default_obtain_base_repo_sources="https://github.com/am11/freebsd-dist/releases/download/11.2/base-i386.txz"
@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 27, 2019

you are right about '-s' options for fetch with github.

I found only one place in CBSD where '-qs' in used:

fetch.subr: [ ${_quiet} -eq 0 ] && _sizebyte=$( ${_FETCH_CMD} -qs ${_url} 2>/dev/null | /usr/bin/awk '{printf $1}' )

https://github.com/cbsd/cbsd/blob/develop/fetch.subr#L43-L54

but it seems it should not lead to a fatal error as we get empty _sizebyte

I would appreciate if you try again with the latest fixes and comment on whether anything has changed. Thank!

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 27, 2019

btw, now CBSD now saves the source for bases/sources (srcup). you can see it via custom display=source args for 'cbsd sources' or 'cbsd bases':
am11_bases

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 27, 2019

Great! I changed --branch master to --branch develop, as the commits went to develop branch.

Fetching still failed for some reason. Attached are the input (fbsd-jailsetup.zip) and output (output.txt) files:

fbsd-jailsetup.zip
output.txt

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 27, 2019

Changing bintray to github URL:

- default_obtain_base_repo_sources="https://bintray.com/am11/freebsd-dist/download_file?file_path=base-11.2-i386.txz"
+ default_obtain_base_repo_sources="https://github.com/am11/freebsd-dist/releases/download/11.2/base-i386.txz"

in the input script succeeded. I wonder why would it have problem with bintray URL, it works fine with normal fetch:

fetch -v https://bintray.com/am11/freebsd-dist/download_file?file_path=base-11.2-i386.txz
resolving server address: bintray.com:443
SSL options: 83004bff
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES128-GCM-SHA256
Certificate subject: /CN=*.bintray.com
Certificate issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=GeoTrust RSA CA 2018
requesting https://bintray.com/am11/freebsd-dist/download_file?file_path=base-11.2-i386.txz
302 redirect to https://dl.bintray.com/am11/freebsd-dist/base-11.2-i386.txz
resolving server address: dl.bintray.com:443
SSL options: 83004bff
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES128-GCM-SHA256
Certificate subject: /CN=*.bintray.com
Certificate issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=GeoTrust RSA CA 2018
requesting https://dl.bintray.com/am11/freebsd-dist/base-11.2-i386.txz
302 redirect to https://d29vzk4ow07wi7.cloudfront.net/9c8be09d549f6365a43f8a86529110c1ebac4263ac357c8aa25b753d65b1460c?response-content-disposition=attachment%3Bfilename%3D%22base-11.2-i386.txz%22&Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6Imh0dHAqOi8vZDI5dnprNG93MDd3aTcuY2xvdWRmcm9udC5uZXQvOWM4YmUwOWQ1NDlmNjM2NWE0M2Y4YTg2NTI5MTEwYzFlYmFjNDI2M2FjMzU3YzhhYTI1Yjc1M2Q2NWIxNDYwYz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPWF0dGFjaG1lbnQlM0JmaWxlbmFtZSUzRCUyMmJhc2UtMTEuMi1pMzg2LnR4eiUyMiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTU0ODYwOTgyNn0sIklwQWRkcmVzcyI6eyJBV1M6U291cmNlSXAiOiIwLjAuMC4wLzAifX19XX0_&Signature=h2aJAf8ylMIHxElbDUQXn4wccZ3zuxXU0WqVyOh59tpMArv3vfHzuyjAwBm4CQSCiCaOvS86F7eor~vlHU8v4MgJJbyEImECMTyKqxnvWS6Atk66mN86ZKVQ9aT9c0Y~y7hIcIRHkVr~NjEw-nHG2r-ZXiw19HGA2QCw5NjSo4uiqklSYUo4OOvTDw2CcfGHcKk~nsaAU0i~0o1TMWXFHLwi-53X6A4OhRR6C6hACxkrP7y0aSufJrvcmTiVnpf9PQnJvFFY8OQjGbLi8AUiOYooV2fZkIeh8BbyVNjL6NyhrQcw8Mn0kmyuZXVcC1ZqnubBnMwauW1gXDk3ZnegGg__&Key-Pair-Id=APKAIFKFWOMXM2UMTSFA
resolving server address: d29vzk4ow07wi7.cloudfront.net:443
SSL options: 83004bff
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES128-GCM-SHA256
Certificate subject: /C=US/ST=Washington/L=Seattle/O=Amazon.com, Inc./CN=*.cloudfront.net
Certificate issuer: /C=US/O=DigiCert Inc/CN=DigiCert Global CA G2
requesting https://d29vzk4ow07wi7.cloudfront.net/9c8be09d549f6365a43f8a86529110c1ebac4263ac357c8aa25b753d65b1460c?response-content-disposition=attachment%3Bfilename%3D%22base-11.2-i386.txz%22&Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6Imh0dHAqOi8vZDI5dnprNG93MDd3aTcuY2xvdWRmcm9udC5uZXQvOWM4YmUwOWQ1NDlmNjM2NWE0M2Y4YTg2NTI5MTEwYzFlYmFjNDI2M2FjMzU3YzhhYTI1Yjc1M2Q2NWIxNDYwYz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPWF0dGFjaG1lbnQlM0JmaWxlbmFtZSUzRCUyMmJhc2UtMTEuMi1pMzg2LnR4eiUyMiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTU0ODYwOTgyNn0sIklwQWRkcmVzcyI6eyJBV1M6U291cmNlSXAiOiIwLjAuMC4wLzAifX19XX0_&Signature=h2aJAf8ylMIHxElbDUQXn4wccZ3zuxXU0WqVyOh59tpMArv3vfHzuyjAwBm4CQSCiCaOvS86F7eor~vlHU8v4MgJJbyEImECMTyKqxnvWS6Atk66mN86ZKVQ9aT9c0Y~y7hIcIRHkVr~NjEw-nHG2r-ZXiw19HGA2QCw5NjSo4uiqklSYUo4OOvTDw2CcfGHcKk~nsaAU0i~0o1TMWXFHLwi-53X6A4OhRR6C6hACxkrP7y0aSufJrvcmTiVnpf9PQnJvFFY8OQjGbLi8AUiOYooV2fZkIeh8BbyVNjL6NyhrQcw8Mn0kmyuZXVcC1ZqnubBnMwauW1gXDk3ZnegGg__&Key-Pair-Id=APKAIFKFWOMXM2UMTSFA
remote size / mtime: 88757260 / 1546733720
download_file?file_path=base-11.2-i386.txz      8% of   84 MB 1343 kBps 00m57s

Edit: could be because fetch computes output file as download_file?file_path=base-11.2-i386.txz, instead of base-11.2-i386.txz?

Edit 2: Yup, seems related to URL with query string. If we provide short URL http://bit.do/base-11-2-i386-txz, pointing to https://bintray.com/am11/freebsd-dist/download_file?file_path=base-11.2-i386.txz, then it works as the GitHub one. Could the implementation be hardened so it parse out the file name (or remove invalid characters if that is the rootcause of failed fetch)?

olevole added a commit that referenced this issue Jan 27, 2019
old behaviour via ARG="${1%%=*}" and VAL="${1##*=}" is not suitable for values
with "=" symbol, e.g:  params="value=2" get 2 for VAL, not value=2.
up to this point we were lucky that there were no such parameters, but
everything changed with the @am11 arrival ;-) who found the situation where
"=" character may be present in value, Issue #367.
So its time to make this method more reliable
@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 27, 2019

@am11 yeah, this is issue with old parsing method via

ARG="${1%%=}"
VAL="${1##
=}"

I fixed it. it was necessary to do it for a long time.

You can try to git pull and try to get back this url (you can use $ver and $arch variables):

default_obtain_base_repo_sources="https://bintray.com/am11/freebsd-dist/download_file?file_path=base-${ver}-${arch}.txz"

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 27, 2019

Thanks for the quick fix @olevole, it worked! ❤️
I noticed that there are two more fetches:

Fetching meta.txz
Fetching packagesite.txz

could you please tell me where are these coming from? I couldn't find it at https://download.freebsd.org/ftp/releases/i386/11.2-RELEASE/. Also, can I override URLs of these files?

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 30, 2019

Thanks. In a private server, I was able to connect to the internet by hard coding an address in /tmp/jail-11i386.jconf:

ip4_addr="192.168.1.100"

However, in the CirrusCI VMs, I tried hardcoded IP, which wasn't easy as every CI job run comes with new IP address and sometimes range too, as well as DHCP, but ping, pkg-install failed: https://cirrus-ci.com/task/5480265578184704.

If you want, you can try to experiment the CI script and test using the following steps:

Note that from the second matrix in .cirrus.yml, all job with abi: freebsd:11:x86:64 succeed as they run directly on FreeBSD amd64 image provided by Google Computer Engine. The i386 jobs are using cbsd jail.

IMO, other users will profit from this work as well, if we could sort out the networking issue and clean up redundant configurations (to reduce noise), in order to create a CI script example for community.

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 31, 2019

Most likely due to the fact that nodeippool is not configured

https://cirrus-ci.com/task/5833571768991744:

   JID  Hostname                      Path
        Name                          State
        CPUSetID
        IP Address(es)
     1  jail-11i386.my.domain         /usr/jails/jail-11i386
        jail-11i386                   ACTIVE
        2
        10.0.0.1
nodeip="auto"
nodeippool="10.0.0.0/24"

i never worked with Cirrus CI ( but this service looks incredible, I like it! ).
What is the IP address usage policy here? We can take any RFC 1918 IPs ?

As i can see, hoster works with 10.x.x.X with prefix 32 network. in other words, we cannot use the same network?

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 31, 2019

Interersting, https://cirrus-ci.com/task/4814732109283328
some times CI/pkg is not work correctly

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 31, 2019

Someone has just updated node.js v11 in pkgs to newver version: 😹
image
http://pkg.freebsd.org/freebsd:11:x86:64/latest/All/

It is bit of a pain that we cannot install specific version of package using pkg-install, as required by this scenario where node-sass is a package with native module, and has to compile against each version of v8 JavaScript engine library (and node.js lacks foreign function interface - FFI). Therefore we have to keep up with full versions for now instead of just the major versions (node11, node10 etc.). I was thinking about creating a shell function to parse out available version from fetch http://pkg.freebsd.org/freebsd:11:x86:64/latest/All/ -o pkgList, but haven't gotten around to it.

Meanwhile, we can update this line: https://github.com/olevole/node-sass/blob/freebsd-ci/.cirrus.yml#L9 to have node-11.8.0 :)

Yup, CirrusCI have recently introduced FreeBSD support (8th Dec. 2018), and I think is the first managed CI system with FreeBSD.

I couldn't find the clear documentation from Cirrus on allowed IP address, could it be somehow dynamically inferred for nodeippool, like Docker and LXC do on Linux? This way the defaults will work on [m]any systems and the (few) advanced users will still have ability to override default.

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 31, 2019

About version: ok, got it. Now settings:

  env:
    matrix:
      - node_js: "11"
        nodeArchive: "node-11.8.0"
        npmArchive: "npm-6.4.1_1"
      - node_js: "10"
        nodeArchive: "node10-10.15.1"
        npmArchive: "npm-node10-6.4.1_1"
      - node_js: "8"
        nodeArchive: "node8-8.15.0"
        npmArchive: "npm-node8-6.4.1_1"
      - node_js: "6"
        nodeArchive: "node6-6.16.0_1"
        npmArchive: "npm-node6-6.4.1_1"

As i can see, CBSD nodeippool and CBSD nat works fine - i can ping 8.8.8.8 from jail.
But there are some features with the resolver ( may be udp/53 rejected to untrusted/external resources )

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 31, 2019

Sorry for not mentioning it early, I had to apply this patch to workaround hostname check: am11@87e17f6 and meant to ask, is this check necessary? But I guess you already have it under control. :)

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 31, 2019

yeah, i've fix this via

fqdn=$( hostname )
hostname=$( hostname -s )

if [ "${fqdn}" = "${hostname}" ]; then
        echo "set hostname $hostname -> ${hostname}.my.domain"
        sysrc hostname="${hostname}.my.domain"
        hostname ${hostname}.my.domain
fi

;)

i don't remember why i had to use FQDN for CBSD.
Most likely for node name uniqueness (when 'cbsd node mode=add' is used)

I think it needs to be fixed and change to UUID.

olevole added a commit that referenced this issue Jan 31, 2019
@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 31, 2019

@am11
maybe we should inherit resolver settings from hoster ? Now resolver work:

https://cirrus-ci.com/task/5351638421209088

  • nc -z google.com 80
    Connection to google.com 80 port [tcp/http] succeeded!

I added:

cbsd jailscp /etc/resolv.conf jail-11i386:/etc/resolv.conf

into configure_freebsd_ci_jail.sh

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 31, 2019

Well, it seems the problem with resolv is solved:

https://cirrus-ci.com/task/5351638421209088

but I don't know where 'Signaled to exit!' comes from. Or just !=0 exit code from last command?

It was my fork from your branch: https://github.com/olevole/node-sass/commits/freebsd-ci

where I added more debug. And looks like key instructions in solving a problem is:

  1. configure nodeippool with
cp /usr/local/cbsd/share/initenv.conf /tmp/initenv.conf
sysrc -qf /tmp/initenv.conf nodeippool=192.168.0.0/24
  1. configure nat and load pf module by hand
cbsd natcfg fw_new=fw natip_new=${auto_iface}
kldload pf || true
cbsd naton

( for some reason it does not load itself. although should )

  1. configure ip forwarding:

sysctl -w net.inet.ip.forwarding=1

  1. inherits resolv.conf from hoster:

cbsd jailscp /etc/resolv.conf jail-11i386:/etc/resolv.conf

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 31, 2019

By the way, it seems to me that cross-build is possible on FreeBSD. I think I did it once.
But I need to recall and check how I did it

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 31, 2019

I have refactored it a bit and removed ping. I am not sure why same sed command works in amd64 but i386 gives this error:

$execPrefix sed -i '' 's/quarterly/latest/g' /etc/pkg/FreeBSD.conf
sed: 1: "/etc/pkg/FreeBSD.conf": extra characters at the end of p command

https://cirrus-ci.com/task/6128895599312896

Is there any escaping required with jexec?

Edit:
Extra -- after -i worked with jexec, and without jexec, -- doesn't work. So I ended up with:

if test "$abi" = "freebsd:11:x86:32"; then
  ./scripts/configure_freebsd_ci_jail.sh $CIRRUS_WORKING_DIR;
  $execPrefix sed -i -- '' 's/quarterly/latest/g' /etc/pkg/FreeBSD.conf; # use latest ports
  $execPrefix pkg update -f;
else
  sed -i '' 's/quarterly/latest/g' /etc/pkg/FreeBSD.conf; # use latest ports
fi
@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 31, 2019

yes, '--' intercepted by CBSD parsing, this is not jexec limitation. You can use HEREDOC here, e.g:

cbsd jexec jname=jail11 /bin/sh << EOF
sed -i '' 's/quarterly/latest/g' /etc/pkg/FreeBSD.conf
/etc/pkg/FreeBSD.conf
pkg update -f
...
EOF

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 31, 2019

Nice to know heredoc also works. I was having some issues with heredoc syntax in .cirrus.yml before, which works with .travis.yml, so I switched to using .sh file. :)
Regarding the internet connectivity in jail, did it ever work? It seems like this debug portion was only jexec'ing the ifconfig etc., but the ping tests and nc command was ran in hoster?

# debug
set -o xtrace
uname -a
cbsd jls
jls -v
cat /etc/resolv.conf
grep nodeip ~cbsd/nc.inventory
cbsd jexec jname=jail-11i386 cat /etc/resolv.conf
cbsd jailscp /etc/resolv.conf jail-11i386:/etc/resolv.conf
cbsd jexec jname=jail-11i386 ifconfig
ping -c1 8.8.8.8 ||true
#nslookup google.com 8.8.8.8 ||true
#dig @8.8.8.8 google.com ||true
nc -z google.com 80 ||true
set +o xtrace
@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 31, 2019

yes, all this command was for hoster -- I tried to find out the server/hoster settings in order to understand what needs to be configured

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 31, 2019

BTW, I definitely like Cirrus CI. a few years ago I really wanted Travis/FreeBSD: travis-ci/travis-ci#6671 but the authors are not very interested in this work.

I will try to study the CCI in more detail. As far cirruslabs/cirrus-ci-docs#125 , jail can provide only i386 arch on x86-64.

Of course, we can use qemu usemode jail for arm/mips jail but it is not serious IMO ;)

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 31, 2019

Just a status update, due to no connectivity in jail, currently $execPrefix pkg update -f step is failing with:

Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD has a wrong packagesite, need to re-create database
pkg: http://pkg.FreeBSD.org/FreeBSD:11:i386/latest/meta.txz: Protocol not supported
repository FreeBSD has no meta file, using default settings
pkg: http://pkg.FreeBSD.org/FreeBSD:11:i386/latest/packagesite.txz: Protocol not supported
Unable to update repository FreeBSD
Error updating repositories!

image

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 31, 2019

oops, this command was for jail:

ping -c1 8.8.8.8 ||true
nc -z google.com 80 ||true

in all likelihood, I was mistaken and forgot to add 'cbsd jexec ' before ;)

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Jan 31, 2019

Ah great, hopefully more CI/CD services will spare love for FreeBSD; appveyor/ci#2844, and unravel the latent potential.. just like Netflix. 😎

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Feb 1, 2019

Cool.

Sorry for by bad with ping/nc ;)
How about this version:

https://github.com/olevole/node-sass/blob/freebsd-ci/scripts/configure_freebsd_ci_jail.sh

no nat, no pf, no interfaces - just use IP address of hoster system

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Feb 1, 2019

Great! 🎉 Here is the simplified version after applying your fixes configure_freebsd_ci_jail.sh and .cirrus.yml.

Is it possible to share $skelDirectory between hoster and jail? I thought setting jailskeldir="$skelDirectory" in jconf will do the mounting/symlinking under the wraps for us and we only need to cd /etc/skel inside the jail. :)

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Feb 1, 2019

You can use fstab.local entry for mount any dirs from hoster inside jail:

https://www.bsdstore.ru/en/12.0.x/wf_jconfig_ssi.html#fstab

For example:

  1. create jail $jail
  2. cat > ~cbsd/jails-fstab/fstab.jail1.local <<EOF
    $skelDirectory /etc/skel nullfs ro 0 0
    EOF

or rw for read-write

  1. restart/start jail

At the moment configure_freebsd_ci_jail.sh used runasap=1 params to atomic jcreate+jstart.
You can back it to runasap=0 (or remove this params, by default = 0) + cbsd jstart. And insert between 'jcreate' and 'jstart' editing of fstab.

Or, you can mount it by hand/command after jail start, from shell:

[ ! -d ~cbsd/jails/jail1/etc/skel ] && mkdir ~cbsd/jails/jail1/etc/skel
mount_nullfs -orw $skelDirectory ~cbsd/jails/jail1/etc/skel     # for RW

or

mount_nullfs -oro $skelDirectory ~cbsd/jails/jail1/etc/skel # for RO

// where jail1 = $jailName

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Feb 1, 2019

For fstab-based version (with some logical fixes) take a look on my merge request:

am11/node-sass#2

@mekanix

This comment has been minimized.

Copy link
Member

@mekanix mekanix commented Feb 7, 2019

I'm sorry I'm late to the party, but would you @am11 be interested in write a doc about it? In any case, THANK YOU SO MUCH for having the patience to fix all the issues with @olevole.

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Feb 10, 2019

@mekanix, thanks. Once 12.0.4 is available in ports (pending approval: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235412), it will simplify the non-interactive jail setup script a bit.

I think non-interactive usage examples could be documented in quickstart guides: https://www.bsdstore.ru/en/cbsd_quickstart.html. I will think of it a bit more and try to send PR at https://github.com/cbsd/cbsd-wwwdoc.

Do you have anything particular in mind that could be documented?


This will probably require a separate RFC post:

It would be great if the new age model of CBSD goes towards simpler (maybe only non-interactive, without tui) command line experience for the repetitive functions: initialize, configure, create, start and execute commands in jail. Something as concise as docker:

  1. install docker using your favorite package manager
  2. start hacking right away: docker run busybox echo "hello world!" (downloads, initializes, configures and executes.. all with one single command)

here some of the configuration, esp. those related to networking and filesystem, are resolved dynamically based on host state and fits the need of many consumers. Advanced users, who require granular control, can also supplement more arguments to the same commands.

We have pretty much achieved the similar thing with a script in this thread, that can be used in scratch VMs and CI jobs: https://github.com/sass/node-sass/blob/fb33306/scripts/configure_freebsd_ci_jail.sh.

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Feb 16, 2019

commited to ports tree: 12.0.4

@am11

This comment has been minimized.

Copy link
Author

@am11 am11 commented Feb 17, 2019

Great, i think it is now a matter of time before it starts showing up in index page http://pkg.freebsd.org/freebsd:11:x86:64/latest/All/. Latest ports are still picking up 12.0.3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.