Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSError: [Errno 30] Read-only file system: '/iocage/jails/x8dtu-pg01/root/etc/resolv.conf' #883

Closed
dlangille opened this issue Feb 28, 2019 · 37 comments

Comments

@dlangille
Copy link
Contributor

commented Feb 28, 2019

NOTE: I now know why the jails were restarting without manual intervention. Not iocage related, but I'll document that later.

Still not sure why they get set RO though.

I reverted to 0.9.10 and jails start. The issue seems related to iocage 1.1

Jails are now back online. No immediate help required, thank you.

$ iocage --version
Version 1.1 RELEASE 2019/01

After rebooting the system all jails seem to be mounted read-only.

Restarting one jail gives:

[dan@x8dtu:~] $ sudo iocage restart x8dtu-pg01
* Stopping x8dtu-pg01
  + Executing prestop OK
  + Stopping services OK
  + Removing devfs_ruleset: 5 OK
  + Removing jail process OK
  + Executing poststop OK
* Starting x8dtu-pg01
  + Started OK
  + Using devfs_ruleset: 5
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage_cli/restart.py", line 41, in cli
    ioc.IOCage(jail=jail).restart(soft)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1432, in restart
    self.start()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1788, in start
    suppress_exception=ignore_exception
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 76, in __init__
    raise e
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 73, in __init__
    self.__start_jail__()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 530, in __start_jail__
    self.start_generate_resolv()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 991, in start_generate_resolv
    shutil.copy(resolver, f"{self.path}/root/etc/resolv.conf")
  File "/usr/local/lib/python3.6/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/lib/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
OSError: [Errno 30] Read-only file system: '/iocage/jails/x8dtu-pg01/root/etc/resolv.conf'

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Feb 28, 2019

Interesting:

[dan@x8dtu:~] $ zfs get -r readonly main_tank/iocage/jails/x8dtu-pg01
NAME                                                             PROPERTY  VALUE   SOURCE
main_tank/iocage/jails/x8dtu-pg01                                readonly  on      local
main_tank/iocage/jails/x8dtu-pg01@Before-py36-iocage             readonly  -       -
main_tank/iocage/jails/x8dtu-pg01@BeforeUpgrade.1017.11.02       readonly  -       -
main_tank/iocage/jails/x8dtu-pg01@before.first.backups           readonly  -       -
main_tank/iocage/jails/x8dtu-pg01/root                           readonly  on      inherited from main_tank/iocage/jails/x8dtu-pg01
main_tank/iocage/jails/x8dtu-pg01/root@Before-py36-iocage        readonly  -       -
main_tank/iocage/jails/x8dtu-pg01/root@BeforeUpgrade.1017.11.02  readonly  -       -
main_tank/iocage/jails/x8dtu-pg01/root@before.first.backups      readonly  -       -
[dan@x8dtu:~] $ 

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

I turn them off:

[dan@x8dtu:~] $ sudo zfs set readonly=off main_tank/iocage/jails/x8dtu-ingress01 main_tank/iocage/jails/x8dtu-nginx01 main_tank/iocage/jails/x8dtu-pg01

They get turned back on later... I don't know why yet.

[dan@x8dtu:~] $ zfs get -r readonly main_tank/iocage/jails | grep 'readonly  on'
main_tank/iocage/jails/x8dtu-ingress01                                  readonly  on      local
main_tank/iocage/jails/x8dtu-ingress01/root                             readonly  on      inherited from main_tank/iocage/jails/x8dtu-ingress01
[dan@x8dtu:~] $ 
@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

I stop all the jails: sudo iocage stop ALL

I confirm they are down: sudo iocage list

[dan@x8dtu:~] $ zfs get -r readonly main_tank/iocage/jails | grep 'readonly  on'
main_tank/iocage/jails/mx-ingress01                                     readonly  on      local
main_tank/iocage/jails/mx-ingress01/root                                readonly  on      inherited from main_tank/iocage/jails/mx-ingress01
main_tank/iocage/jails/x8dtu-ingress01                                  readonly  on      local
main_tank/iocage/jails/x8dtu-ingress01/root                             readonly  on      inherited from main_tank/iocage/jails/x8dtu-ingress01
main_tank/iocage/jails/x8dtu-nginx01                                    readonly  on      local
main_tank/iocage/jails/x8dtu-nginx01/root                               readonly  on      inherited from main_tank/iocage/jails/x8dtu-nginx01
main_tank/iocage/jails/x8dtu-pg01                                       readonly  on      local
main_tank/iocage/jails/x8dtu-pg01/root                                  readonly  on      inherited from main_tank/iocage/jails/x8dtu-pg01
[dan@x8dtu:~] $ 

Turn readonly off:

[dan@x8dtu:~] $ sudo zfs set readonly=off main_tank/iocage/jails/x8dtu-ingress01 main_tank/iocage/jails/x8dtu-nginx01 main_tank/iocage/jails/x8dtu-pg01 main_tank/iocage/jails/mx-ingress01ingress01

verify it's off:

[dan@x8dtu:~] $ zfs get -r readonly main_tank/iocage/jails | grep 'readonly  on'
[dan@x8dtu:~] $ 

Wait.

Jails are now running again. and datasets are RO

WTF?

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

To track this down, I've done this before going out:

sudo iocage stop ALL
sudo zfs set readonly=off main_tank/iocage/jails/x8dtu-ingress01 main_tank/iocage/jails/x8dtu-nginx01 main_tank/iocage/jails/x8dtu-pg01 `main_tank/iocage/jails/mx-ingress01`
sudo service nrpe3 stop
sudo service bacula-fd stop
sudo service snmpd stop
sudo service postfix stop

While I've been gone, the jails have stayed stopped and the readonly has not returned.

[dan@x8dtu:~] $ jls
   JID  IP Address      Hostname                      Path
[dan@x8dtu:~] $ 
[dan@x8dtu:~] $ zfs get -r readonly main_tank/iocage/jails | grep 'readonly  on'
[dan@x8dtu:~] $ 

I also commented this entry out from /etc/crontab:

#*/3     *       *       *       *       root    /usr/local/etc/snmp/smart -u
@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

I don't think it's Postfix, so I'll start that back up.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

Let's try this:

[dan@x8dtu:/usr/local/sbin] $ sudo iocage start x8dtu-pg02
* Starting x8dtu-pg02_vpn_unixathome_org
  + Started OK
  + Using devfs_ruleset: 5
  + Starting services OK
  + Executing poststart OK
[dan@x8dtu:/usr/local/sbin] $ 
@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

WTF?

[dan@x8dtu:/usr/local/sbin] $ zfs get -r readonly main_tank/iocage/jails | grep 'readonly  on'
[dan@x8dtu:/usr/local/sbin] $ sudo iocage start mx-ingress01
* Starting mx-ingress01
  + Started OK
  + Using devfs_ruleset: 6
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage_cli/start.py", line 60, in cli
    ioc.IOCage(jail=jail, rc=rc).start(ignore_exception=ignore)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1788, in start
    suppress_exception=ignore_exception
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 76, in __init__
    raise e
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 73, in __init__
    self.__start_jail__()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 530, in __start_jail__
    self.start_generate_resolv()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 991, in start_generate_resolv
    shutil.copy(resolver, f"{self.path}/root/etc/resolv.conf")
  File "/usr/local/lib/python3.6/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/lib/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
OSError: [Errno 30] Read-only file system: '/iocage/jails/mx-ingress01/root/etc/resolv.conf'
[dan@x8dtu:/usr/local/sbin] $ zfszfs get -r readonly main_tank/iocage/jails | grep 'readonly  on'
main_tank/iocage/jails/mx-ingress01                                     readonly  on      local
main_tank/iocage/jails/mx-ingress01/root                                readonly  on      inherited from main_tank/iocage/jails/mx-ingress01
main_tank/iocage/jails/x8dtu-ingress01                                  readonly  on      local
main_tank/iocage/jails/x8dtu-ingress01/root                             readonly  on      inherited from main_tank/iocage/jails/x8dtu-ingress01
main_tank/iocage/jails/x8dtu-nginx01                                    readonly  on      local
main_tank/iocage/jails/x8dtu-nginx01/root                               readonly  on      inherited from main_tank/iocage/jails/x8dtu-nginx01
main_tank/iocage/jails/x8dtu-pg01                                       readonly  on      local
main_tank/iocage/jails/x8dtu-pg01/root                                  readonly  on      inherited from main_tank/iocage/jails/x8dtu-pg01
[dan@x8dtu:/usr/local/sbin] $ 

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

[dan@x8dtu:~] $ sudo zpool scrub main_tank
[dan@x8dtu:/var/log] $ zpool status
  pool: main_tank
 state: ONLINE
  scan: scrub in progress since Fri Mar  1 02:48:16 2019
	206M scanned out of 874G at 2.78M/s, 89h21m to go
        0 repaired, 0.02% done
config:

	NAME        STATE     READ WRITE CKSUM
	main_tank   ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    ada0p1  ONLINE       0     0     0
	    ada1p1  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 0h2m with 0 errors on Sat Feb 23 03:04:11 2019
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    ada2p3  ONLINE       0     0     0
	    ada3p3  ONLINE       0     0     0

errors: No known data errors
[dan@x8dtu:/var/log] $ 


@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

Oh this was interesting:

[dan@x8dtu:/usr/local/sbin] $ iocage get all mx-ingress01 | grep -i read
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage_cli/get.py", line 107, in cli
    props = ioc.IOCage(jail=jail, skip_jails=True).get(prop)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1174, in get
    props = ioc_json.IOCJson(path).json_get_value(prop)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 1514, in json_get_value
    return self.get_full_config()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 1165, in get_full_config
    conf, write = self.json_load()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 1487, in json_load
    conf = self.check_config(conf)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 702, in check_config
    exception=ioc_exceptions.CommandNeedsRoot)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 100, in logit
    callback(content, exception)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 70, in callback
    raise callback_exception(message)
iocage_lib.ioc_exceptions.CommandNeedsRoot: You need to be root to convert the configurations to the new format!
[dan@x8dtu:/usr/local/sbin] $ 

Let's try root:

[dan@x8dtu:/usr/local/sbin] $ sudo iocage get all mx-ingress01 | grep -i read
available:readonly
compressratio:readonly
mountpoint:readonly
origin:readonly
used:readonly
[dan@x8dtu:/usr/local/sbin] $ zfs get -r readonly main_tank/iocage/jails | grep 'readonly  on'
main_tank/iocage/jails/mx-ingress01                                     readonly  on      local
main_tank/iocage/jails/mx-ingress01/root                                readonly  on      inherited from main_tank/iocage/jails/mx-ingress01
[dan@x8dtu:/usr/local/sbin] $ ≈

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

Looking at zpool history main_tank I see:

2019-03-01.03:10:42 zfs set readonly=off main_tank/iocage/jails/x8dtu-ingress01 main_tank/iocage/jails/x8dtu-nginx01 main_tank/iocage/jails/x8dtu-pg01 main_tank/iocage/jails/mx-ingress01
2019-03-01.03:29:01 zfs set readonly=off main_tank/iocage/jails/x8dtu-pg01
2019-03-01.03:29:03 zfs set readonly=on main_tank/iocage/jails/x8dtu-pg01
2019-03-01.03:29:05 zfs set readonly=off main_tank/iocage/jails/x8dtu-pg01
2019-03-01.03:29:08 zfs set readonly=on main_tank/iocage/jails/x8dtu-pg01
2019-03-01.03:29:10 zfs set readonly=off main_tank/iocage/jails/x8dtu-pg01
2019-03-01.03:29:13 zfs set readonly=on main_tank/iocage/jails/x8dtu-pg01
2019-03-01.03:29:16 zfs set readonly=off main_tank/iocage/jails/x8dtu-pg01
2019-03-01.03:29:22 zfs set readonly=on main_tank/iocage/jails/x8dtu-pg01
2019-03-01.03:29:46 zfs set readonly=off main_tank/iocage/jails/x8dtu-pg01
2019-03-01.03:29:52 zfs set readonly=on main_tank/iocage/jails/x8dtu-pg01
2019-03-01.03:30:10 zfs set readonly=off main_tank/iocage/jails/x8dtu-ingress01 main_tank/iocage/jails/x8dtu-nginx01 main_tank/iocage/jails/x8dtu-pg01 main_tank/iocage/jails/mx-ingress01

[dan@x8dtu:/var/log] $ 

This corresponds nicely with these entries from /var/log/iocage.log

2019/03/01 03:29:02 (ERROR) x8dtu-pg01 is not running!
2019/03/01 03:29:12 (INFO) * Starting x8dtu-pg01
2019/03/01 03:29:12 (INFO)   + Started OK
2019/03/01 03:29:12 (INFO)   + Using devfs_ruleset: 6
2019/03/01 03:29:47 (INFO) * Stopping x8dtu-pg01
2019/03/01 03:29:47 (INFO)   + Executing prestop OK
2019/03/01 03:29:48 (INFO)   + Stopping services OK
2019/03/01 03:29:48 (INFO)   + Removing devfs_ruleset: 6 OK
2019/03/01 03:29:48 (INFO)   + Removing jail process OK
2019/03/01 03:29:48 (INFO)   + Executing poststop OK
@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

Oh, new format?

[dan@x8dtu:/usr/local/sbin] $ iocage list
You need to be root to convert the configurations to the new format!
You need to be root to convert the configurations to the new format!
You need to be root to convert the configurations to the new format!
You need to be root to convert the configurations to the new format!
+-----+-------------------------------+---------+--------------+------------------------------------------+
| JID |             NAME              |  STATE  |   RELEASE    |                   IP4                    |
+=====+===============================+=========+==============+==========================================+
| 38  | mailjail                      | up      | 11.2-RELEASE | [redacted] |
+-----+-------------------------------+---------+--------------+------------------------------------------+
| -   | mx-ingress01                  | CORRUPT | N/A          | N/A                                      |
+-----+-------------------------------+---------+--------------+------------------------------------------+
| -   | x8dtu-ingress01               | CORRUPT | N/A          | N/A                                      |
+-----+-------------------------------+---------+--------------+------------------------------------------+
| -   | x8dtu-nginx01                 | CORRUPT | N/A          | N/A                                      |
+-----+-------------------------------+---------+--------------+------------------------------------------+
| -   | x8dtu-pg01                    | CORRUPT | N/A          | N/A                                      |
+-----+-------------------------------+---------+--------------+------------------------------------------+
| -   | x8dtu-pg02.vpn.unixathome.org | down    | 11.2-RELEASE | [redacted]                 |
+-----+-------------------------------+---------+--------------+------------------------------------------+
[dan@x8dtu:/usr/local/sbin] $ sudo iocage list
+-----+-------------------------------+-------+--------------+------------------------------------------+
| JID |             NAME              | STATE |   RELEASE    |                   IP4                    |
+=====+===============================+=======+==============+==========================================+
| 38  | mailjail                      | up    | 11.2-RELEASE | [redacted] |
+-----+-------------------------------+-------+--------------+------------------------------------------+
| -   | mx-ingress01                  | down  | 11.2-RELEASE | [redacted]             |
+-----+-------------------------------+-------+--------------+------------------------------------------+
| -   | x8dtu-ingress01               | down  | 11.2-RELEASE | [redacted]                 |
+-----+-------------------------------+-------+--------------+------------------------------------------+
| -   | x8dtu-nginx01                 | down  | 11.2-RELEASE | [redacted]                 |
+-----+-------------------------------+-------+--------------+------------------------------------------+
| -   | x8dtu-pg01                    | down  | 11.2-RELEASE | [redacted]     |
+-----+-------------------------------+-------+--------------+------------------------------------------+
| -   | x8dtu-pg02.vpn.unixathome.org | down  | 11.2-RELEASE | [redacted]                 |
+-----+-------------------------------+-------+--------------+------------------------------------------+
[dan@x8dtu:/usr/local/sbin] $ 


1 - the word 'convert' does not appear in man iocage ---- it does, once, but not near the command I think you should refer to: migrate.

2 - those spurious errors when not running as root will lead people down a rabbit hole when they are trying to debug a problem

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 1, 2019

I tried starting another jail:

[dan@x8dtu:/usr/local/sbin] $ sudo iocage start mx-ingress01
* Starting mx-ingress01
  + Started OK
  + Using devfs_ruleset: 6
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage_cli/start.py", line 60, in cli
    ioc.IOCage(jail=jail, rc=rc).start(ignore_exception=ignore)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1788, in start
    suppress_exception=ignore_exception
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 76, in __init__
    raise e
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 73, in __init__
    self.__start_jail__()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 530, in __start_jail__
    self.start_generate_resolv()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 991, in start_generate_resolv
    shutil.copy(resolver, f"{self.path}/root/etc/resolv.conf")
  File "/usr/local/lib/python3.6/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/lib/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
OSError: [Errno 30] Read-only file system: '/iocage/jails/mx-ingress01/root/etc/resolv.conf'

From the log file:

2019/03/01 03:39:48 (INFO) * Starting mx-ingress01
2019/03/01 03:39:48 (INFO)   + Started OK
2019/03/01 03:39:48 (INFO)   + Using devfs_ruleset: 6
2019/03/01 03:40:16 (INFO) * Stopping mx-ingress01
2019/03/01 03:40:16 (INFO)   + Executing prestop OK
2019/03/01 03:40:21 (INFO)   + Stopping services OK
2019/03/01 03:40:21 (INFO)   + Removing devfs_ruleset: 6 OK
2019/03/01 03:40:21 (INFO)   + Removing jail process OK
2019/03/01 03:40:21 (INFO)   + Executing poststop OK

From zpool history:

2019-03-01.03:39:42 zfs set readonly=off main_tank/iocage/jails/mx-ingress01
2019-03-01.03:39:44 zfs set readonly=on main_tank/iocage/jails/mx-ingress01
2019-03-01.03:39:47 zfs set readonly=off main_tank/iocage/jails/mx-ingress01
2019-03-01.03:39:49 zfs set readonly=on main_tank/iocage/jails/mx-ingress01
2019-03-01.03:39:52 zfs set readonly=off main_tank/iocage/jails/mx-ingress01
2019-03-01.03:39:54 zfs set readonly=on main_tank/iocage/jails/mx-ingress01
2019-03-01.03:40:14 zfs set readonly=off main_tank/iocage/jails/mx-ingress01
2019-03-01.03:40:21 zfs set readonly=on main_tank/iocage/jails/mx-ingress01

@sonicaj sonicaj self-assigned this Mar 4, 2019

@sonicaj

This comment has been minimized.

Copy link
Member

commented Mar 5, 2019

Hello @dlangille could you please provide me with an output of zfs list ?

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 5, 2019

$ zfs list
NAME                                                        USED  AVAIL  REFER  MOUNTPOINT
main_tank                                                   874G  3.54T    96K  /main_tank
main_tank/data                                              786G  3.54T    88K  /main_tank/data
main_tank/data/dvl                                          755G  3.54T   755G  /main_tank/data/dvl
main_tank/data/freshports                                  30.6G  3.54T    88K  /main_tank/data/freshports
main_tank/data/freshports/backend                          2.86G  3.54T    88K  /main_tank/data/freshports/backend
main_tank/data/freshports/backend/cache                    1.08M  3.54T    96K  /iocage/jails/x8dtu-ingress01/root/var/db/freshports/cache
main_tank/data/freshports/backend/cache/html                924K  3.54T   924K  /iocage/jails/x8dtu-ingress01/root/var/db/freshports/cache/html
main_tank/data/freshports/backend/cache/spooling             88K  3.54T    88K  /iocage/jails/x8dtu-ingress01/root/var/db/freshports/cache/spooling
main_tank/data/freshports/backend/queues                   2.86G  3.54T    88K  none
main_tank/data/freshports/backend/queues/archive           2.85G  3.54T  2.85G  /iocage/jails/x8dtu-ingress01/root/var/db/freshports/message-queues/archive
main_tank/data/freshports/backend/queues/incoming            88K  3.54T    88K  /iocage/jails/x8dtu-ingress01/root/var/db/ingress/message-queues/incoming
main_tank/data/freshports/backend/queues/recent            6.81M  3.54T  6.81M  /iocage/jails/x8dtu-ingress01/root/var/db/freshports/message-queues/recent
main_tank/data/freshports/backend/queues/retry              208K  3.54T   208K  /iocage/jails/x8dtu-ingress01/root/var/db/freshports/message-queues/retry
main_tank/data/freshports/ports-jail                       25.5G  3.54T   184K  /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail
main_tank/data/freshports/ports-jail/var                   25.5G  3.54T    88K  /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var
main_tank/data/freshports/ports-jail/var/db                25.5G  3.54T    88K  /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db
main_tank/data/freshports/ports-jail/var/db/repos          25.5G  3.54T  25.5G  /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db/repos
main_tank/data/freshports/website                          2.27G  3.54T    88K  none
main_tank/data/freshports/website/cache                    2.27G  3.54T   128M  /iocage/jails/x8dtu-nginx01/root/var/db/freshports/cache
main_tank/data/freshports/website/cache/daily              15.1M  3.54T  15.1M  /iocage/jails/x8dtu-nginx01/root/var/db/freshports/cache/daily
main_tank/data/freshports/website/cache/news               3.10M  3.54T  3.10M  /iocage/jails/x8dtu-nginx01/root/var/db/freshports/cache/news
main_tank/data/freshports/website/cache/pages                88K  3.54T    88K  /iocage/jails/x8dtu-nginx01/root/var/db/freshports/cache/pages
main_tank/data/freshports/website/cache/ports              2.12G  3.54T  2.12G  /iocage/jails/x8dtu-nginx01/root/var/db/freshports/cache/ports
main_tank/data/freshports/website/cache/spooling            120K  3.54T   120K  /iocage/jails/x8dtu-nginx01/root/var/db/freshports/cache/spooling
main_tank/home                                             10.5G  3.54T  10.5G  /usr/home
main_tank/iocage                                           78.0G  3.54T   116K  /iocage
main_tank/iocage/download                                   503M  3.54T    96K  /iocage/download
main_tank/iocage/download/11.0-RELEASE                      112M  3.54T   112M  /iocage/download/11.0-RELEASE
main_tank/iocage/download/11.1-RELEASE                      119M  3.54T   119M  /iocage/download/11.1-RELEASE
main_tank/iocage/download/11.2-RELEASE                      272M  3.54T   272M  /iocage/download/11.2-RELEASE
main_tank/iocage/images                                      96K  3.54T    96K  /iocage/images
main_tank/iocage/jails                                     75.9G  3.54T   104K  /iocage/jails
main_tank/iocage/jails/mailjail                             438M  3.54T    92K  /iocage/jails/mailjail
main_tank/iocage/jails/mailjail/root                        438M  3.54T  1.35G  /iocage/jails/mailjail/root
main_tank/iocage/jails/mx-ingress01                        1.63G  3.54T   100K  /iocage/jails/mx-ingress01
main_tank/iocage/jails/mx-ingress01/root                   1.63G  3.54T  1.38G  /iocage/jails/mx-ingress01/root
main_tank/iocage/jails/x8dtu-ingress01                     2.98G  3.54T   116K  /iocage/jails/x8dtu-ingress01
main_tank/iocage/jails/x8dtu-ingress01/root                2.98G  3.54T  1.86G  /iocage/jails/x8dtu-ingress01/root
main_tank/iocage/jails/x8dtu-nginx01                       4.16G  3.54T   116K  /iocage/jails/x8dtu-nginx01
main_tank/iocage/jails/x8dtu-nginx01/root                  4.16G  3.54T  2.11G  /iocage/jails/x8dtu-nginx01/root
main_tank/iocage/jails/x8dtu-pg01                          38.6G  3.54T   100K  /iocage/jails/x8dtu-pg01
main_tank/iocage/jails/x8dtu-pg01/root                     38.6G  3.54T  31.4G  /iocage/jails/x8dtu-pg01/root
main_tank/iocage/jails/x8dtu-pg02.vpn.unixathome.org       28.1G  3.54T    92K  /iocage/jails/x8dtu-pg02.vpn.unixathome.org
main_tank/iocage/jails/x8dtu-pg02.vpn.unixathome.org/root  28.1G  3.54T  29.0G  /iocage/jails/x8dtu-pg02.vpn.unixathome.org/root
main_tank/iocage/log                                        332K  3.54T   156K  /iocage/log
main_tank/iocage/releases                                  1.59G  3.54T    96K  /iocage/releases
main_tank/iocage/releases/11.0-RELEASE                      355M  3.54T    96K  /iocage/releases/11.0-RELEASE
main_tank/iocage/releases/11.0-RELEASE/root                 355M  3.54T   355M  /iocage/releases/11.0-RELEASE/root
main_tank/iocage/releases/11.1-RELEASE                      305M  3.54T    88K  /iocage/releases/11.1-RELEASE
main_tank/iocage/releases/11.1-RELEASE/root                 305M  3.54T   302M  /iocage/releases/11.1-RELEASE/root
main_tank/iocage/releases/11.2-RELEASE                      971M  3.54T    88K  /iocage/releases/11.2-RELEASE
main_tank/iocage/releases/11.2-RELEASE/root                 971M  3.54T   971M  /iocage/releases/11.2-RELEASE/root
main_tank/iocage/templates                                   96K  3.54T    96K  /iocage/templates
main_tank/snapshots-for-backup                               88K  3.54T    88K  /main_tank/snapshots-for-backup
zroot                                                      28.1G  77.5G    96K  /zroot
zroot/ROOT                                                 4.04G  77.5G    96K  none
zroot/ROOT/11.1-RELEASE                                       8K  77.5G  2.29G  /
zroot/ROOT/default                                         4.04G  77.5G  2.85G  /
zroot/TESTING                                               616K  77.5G    88K  /zroot/TESTING
zroot/TESTING/top                                           528K  77.5G    88K  /zroot/TESTING/top
zroot/TESTING/top/test1                                      88K  77.5G    88K  /zroot/TESTING/top/test1
zroot/TESTING/top/test2                                      88K  77.5G    88K  /zroot/TESTING/top/test2
zroot/TESTING/top/test3                                      88K  77.5G    88K  /zroot/TESTING/top/test3
zroot/TESTING/top/test4                                      88K  77.5G    88K  /zroot/TESTING/top/test4
zroot/TESTING/top/test5                                      88K  77.5G    88K  /zroot/TESTING/top/test5
zroot/data                                                 17.2G  77.5G    88K  /zroot/data
zroot/data/dan-pg                                            88K  77.5G    88K  /usr/home/dan/pg
zroot/data/freshports                                      3.37G  77.5G    88K  none
zroot/data/freshports/repo                                 3.37G  77.5G    88K  none
zroot/data/freshports/repo/PORTS-head                      3.37G  77.5G  3.37G  /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db/repos/PORTS-head
zroot/data/ingress01-testing                                675M  77.5G   675M  /iocage/jails/x8dtu-ingress01/root/usr/home/dan/tmp-fast
zroot/data/pg01-postgres                                   13.2G  77.5G  13.2G  /iocage/jails/x8dtu-pg01/root/var/db/postgres
zroot/tmp                                                   216K  77.5G   120K  /tmp
zroot/usr                                                   192K  77.5G    96K  /usr
zroot/usr/src                                                96K  77.5G    96K  /usr/src
zroot/var                                                  6.75G  77.5G    96K  /var
zroot/var/audit                                              96K  77.5G    96K  /var/audit
zroot/var/crash                                              96K  77.5G    96K  /var/crash
zroot/var/log                                              6.75G  77.5G  6.75G  /var/log
zroot/var/mail                                              184K  77.5G   120K  /var/mail
zroot/var/tmp                                               968K  77.5G   760K  /var/tmp
$ 

@sonicaj

This comment has been minimized.

Copy link
Member

commented Mar 5, 2019

@dlangille could you please verify if reproduction steps i tried are valid:

  1. install iocage 0.9
  2. create jails ( jails + template jails )
  3. start a few
  4. upgrade iocage to 1.1
  5. Restart/start all jails, try individually the same process
  6. No reproduction case

Could you please let me know if i perhaps missed something ?
I am wondering if this might be isolated to your environment - initially i thought it might be related to template jails where iocage might be going rogue setting all jails to readonly, but. that seems to not be the case.
If you think the steps are accurate, could you please email me a debug
iocage debug
to waqar (at ) ixsystems ( dot ) com ?
Thank you

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 5, 2019

FYI, after the upgrade to iocage 1.1 I restarted the system.

Another server of mine is running iocage 1.1 fine. It was upgraded at the same time.

You want me to run iocage debug using iocage 1.1?

@sonicaj

This comment has been minimized.

Copy link
Member

commented Mar 5, 2019

@dlangille yes please, i forgot to add reboot, it didn't help as well and jails start as desired :)

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 5, 2019

There is no output from iocage debug

@sonicaj

This comment has been minimized.

Copy link
Member

commented Mar 5, 2019

It would be under /iocage/ for you - a folder with debug as it's name

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 5, 2019

FYI, after running debug:

[dan@x8dtu:/iocage/debug] $ zfs get -r readonly main_tank/iocage/jails | grep 'readonly  on'
main_tank/iocage/jails/mx-ingress01                                     readonly  on      local
main_tank/iocage/jails/mx-ingress01/root                                readonly  on      inherited from main_tank/iocage/jails/mx-ingress01
main_tank/iocage/jails/x8dtu-ingress01                                  readonly  on      local
main_tank/iocage/jails/x8dtu-ingress01/root                             readonly  on      inherited from main_tank/iocage/jails/x8dtu-ingress01
main_tank/iocage/jails/x8dtu-nginx01                                    readonly  on      local
main_tank/iocage/jails/x8dtu-nginx01/root                               readonly  on      inherited from main_tank/iocage/jails/x8dtu-nginx01
main_tank/iocage/jails/x8dtu-pg01                                       readonly  on      local
main_tank/iocage/jails/x8dtu-pg01/root                                  readonly  on      inherited from main_tank/iocage/jails/x8dtu-pg01
[dan@x8dtu:/iocage/debug] $ 

Nothing new in /var/log/iocage.log

Normal service by jails stopped.

I reinstalled 0.9.0, stopped jails, set readonly back, started jails. Service resumed.

debug output emailed as requested.

@sonicaj

This comment has been minimized.

Copy link
Member

commented Mar 6, 2019

@dlangille can you please confirm if you changed the template property of the jails in the debug ? It is set to none, whereas it should be no. The issue is resolved in the latest master - however i would like to know if we at some point perhaps set the template property to none. We should have raised an exception if you tried to set it to some value which we don't recognise i.e none, but that too is resolved in the latest master.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 6, 2019

I cannot say for sure, but I do not recall anything like that.

Searching both my blog (https://dan.langille.org/) and https://gist.github.com I found nothing to indicate I might have done that.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 6, 2019

FYI: this zpool has been around since Wed Apr 12 21:36 2017

Looking at https://www.freshports.org/sysutils/iocage/ this host started with the old non-python version of iocage, probably 1.7.6

The zfs datasets were created in Oct 2017

[dan@x8dtu:/iocage/debug] $ zfs get creation main_tank/iocage/jails/mx-ingress01 main_tank/iocage/jails/x8dtu-ingress01 main_tank/iocage/jails/x8dtu-nginx01 main_tank/iocage/jails/x8dtu-pg01
NAME                                    PROPERTY  VALUE                  SOURCE
main_tank/iocage/jails/mx-ingress01     creation  Wed Oct  4 21:47 2017  -
main_tank/iocage/jails/x8dtu-ingress01  creation  Wed Oct  4 21:47 2017  -
main_tank/iocage/jails/x8dtu-nginx01    creation  Wed Oct  4 21:47 2017  -
main_tank/iocage/jails/x8dtu-pg01       creation  Wed Oct  4 21:47 2017  -
[dan@x8dtu:/iocage/debug] $ 

By then, the new Python iocage was in place: 0.9.9.1_1

Searching issues here, I find #374 where I was using 0.9.5 which means the Python version

At that issue, I find this gist, https://gist.github.com/dlangille/142607fabf5ba64ff84f13e4910817d9

That gist shows the host x8dtu, the one in question for this issue.

@sonicaj

This comment has been minimized.

Copy link
Member

commented Mar 6, 2019

@dlangille this is probably a leftover from the legacy then as we haven't done this with the python version ( i'll double check though ). Thank you for walking with me on this one. I'll be adding a fix for this and should be available in the master tomorrow

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 6, 2019

Thanks. We should also ping the FreeBSD maintainer for a new release?

Let me know if you'd rather I test this before that.

@skarekrow

This comment has been minimized.

Copy link
Member

commented Mar 6, 2019

@sonicaj

This comment has been minimized.

Copy link
Member

commented Mar 7, 2019

@dlangille this has already been sorted out when we introduced boolean props in master. So an upgrade will have this covered. Thanks again!

@sonicaj sonicaj closed this Mar 7, 2019

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 7, 2019

@skarekrow I shudder at upgrades which can't be backed out.

All assurances aside, if it fails to fix the problem, FreshPorts would be offline until it is fixed. :/

Are snapshots feasible for saving this?

@skarekrow

This comment has been minimized.

Copy link
Member

commented Mar 7, 2019

@dlangille Well in this case it actually makes life super great for users and us. Everything that is truthy can be specified as no/yes, off/on, 1/0, true/false with no care about the odd differences. So that ondisk migration is not backwards compatible.

I would snapshot your jails dataset yes, then you can just roll back each jails configuration. Nothing inside the jails change.

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 7, 2019

So I just need to restore each /iocage/jails/$JAILNAME/config.json file?

@skarekrow

This comment has been minimized.

Copy link
Member

commented Mar 7, 2019

If you run into any regressions from the latest Git code, yep :)

@skarekrow skarekrow added bug and removed Work in Progress labels Mar 8, 2019

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 18, 2019

Should I be using .0.0.20190308,1 from https://www.freshports.org/sysutils/iocage-devel/ ?

I ask because I've been watching the non-devel port.

@skarekrow

This comment has been minimized.

Copy link
Member

commented Mar 18, 2019

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Mar 22, 2019

the devel port is borked. No commands. See #894

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Apr 9, 2019

I suspect this bug just took FreshPorts offline. Any idea on a working release please?

@dlangille

This comment has been minimized.

Copy link
Contributor Author

commented Apr 9, 2019

"solution": install git

@muahdib69

This comment has been minimized.

Copy link

commented May 27, 2019

This happens to me to, Upgraded yesterday to 12.0 on my server, I now have py36-iocage-1.1 installed.

Some jails are fine, others have exactly the same behavior, zfs set to readonly and iocage can not create the temporary directory or file to create the resolv.conf and breaks out with error, jail is up but readonly.

I tried to install de development version (thanks @dlangille for the tip on install git!!!) but it is the same.

I checked with different jails I had, and I did see that the jails that are running did have template=0, versus my other jails which did not start did have template=1.

Using iocage set template=0 JAILNAME did result in the message:
"JAILNAME is already a jail!"

going into the jail directory and changing template from 1 to 0 did the trick and now starting the jail it startsup without read-only set on the file system.

It seems to me that there is some confusion with template jails, maybe jails created from templates have some upgrade issues in the 1.1 version.

best
Ray

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.