Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fault injection with replace operation with no-op #18

Closed
ligurio opened this issue Dec 2, 2020 · 1 comment
Closed

Fault injection with replace operation with no-op #18

ligurio opened this issue Dec 2, 2020 · 1 comment

Comments

@ligurio
Copy link
Owner

ligurio commented Dec 2, 2020

Like https://github.com/stewartsmith/libeatmydata

replace operation with no-op for those operations where it possible

Implementation https://github.com/stewartsmith/libeatmydata/blob/master/libeatmydata/libeatmydata.c

@ligurio ligurio mentioned this issue Dec 2, 2020
9 tasks
@ligurio ligurio changed the title Replace fsync(2) with no-op Fault injection with replace fsync(2) with no-op Jan 27, 2021
ligurio added a commit that referenced this issue Apr 15, 2021
unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

Supported error injections:

- errinj_errno
- errinj_kill_caller
- errinj_noop
- errinj_remove_target
- errinj_sleep_op
- errinj_truncate_target

TODO:
  - mandoc: update unreliablefs.conf.5, see [2]
  - mandoc: update unreliablefs.1
  - mandoc: describe regex options (REG_ICASE | REG_EXTENDED)
  - mandoc: describe limitations (PATH_MAX)
  - learn how to read global fuse configuration (seed, base_dir)
  - add random errno support
  - send specified signal and SIGKILL by default

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html
3. https://github.com/mariusae/trickle/blob/master/trickled.conf.5
4. libfuse/libfuse#155

Closes #3
Closes #4
Closes #56

Closes #35
Closes #34
Closes #28
Closes #6
Closes #9
Closes #18
Closes #29
ligurio added a commit that referenced this issue Apr 15, 2021
unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

Supported error injections:

- errinj_errno
- errinj_kill_caller
- errinj_noop
- errinj_remove_target
- errinj_sleep_op
- errinj_truncate_target

TODO:
  - mandoc: update unreliablefs.conf.5, see [2]
  - mandoc: update unreliablefs.1
  - mandoc: describe regex options (REG_ICASE | REG_EXTENDED)
  - mandoc: describe limitations (PATH_MAX)
  - learn how to read global fuse configuration (seed, base_dir)
  - add random errno support
  - send specified signal and SIGKILL by default

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html
3. https://github.com/mariusae/trickle/blob/master/trickled.conf.5
4. libfuse/libfuse#155

Closes #3 (configuration support)
Closes #4 (better randomization)
Closes #35 (remove file)
Closes #34 (truncate file)
Closes #28 (kill caller)
Closes #6 (realistic errno's)
Closes #9 (set custom errno)
Closes #18 (no-op)
Closes #29 (delayed op)
ligurio added a commit that referenced this issue Apr 18, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop".
More error injections are coming.

TODO:

- print messages only when debug mode is on
- destroy_config()
- read_config()
- tests?

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html
3. https://github.com/libfuse/libfuse/wiki/Option-Parsing

Closes #3
Closes #18

Closes #60
Closes #12
Closes #7
Closes #1
ligurio added a commit that referenced this issue Apr 19, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop".
More error injections are coming.

TODO:

- print messages only when debug mode is on
- destroy_config()
- read_config()
- tests?

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html
3. https://github.com/libfuse/libfuse/wiki/Option-Parsing

Closes #3
Closes #18

Closes #60
Closes #12
Closes #7
Closes #1
ligurio added a commit that referenced this issue Apr 19, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop".
More error injections are coming.

TODO:

- print messages only when debug mode is on
- destroy_config()
- read_config()
- tests?

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html
3. https://github.com/libfuse/libfuse/wiki/Option-Parsing

Closes #3
Closes #18

Closes #60
Closes #12
Closes #7
Closes #1
ligurio added a commit that referenced this issue Apr 19, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop".
More error injections are coming.

TODO:

- print messages only when debug mode is on
- destroy_config()
- read_config()

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html
3. https://github.com/libfuse/libfuse/wiki/Option-Parsing

Closes #3
Closes #18

Closes #60
Closes #12
Closes #7
Closes #1
ligurio added a commit that referenced this issue Apr 20, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop".
More error injections are coming.

TODO:

- print messages only when debug mode is on
- destroy_config()
- config read in main() is lost

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html
3. https://github.com/libfuse/libfuse/wiki/Option-Parsing

Closes #3
Closes #18

Closes #60
Closes #12
Closes #7
Closes #1
@ligurio ligurio changed the title Fault injection with replace fsync(2) with no-op Fault injection with replace operation with no-op Apr 20, 2021
ligurio added a commit that referenced this issue Apr 20, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop".
More error injections are coming.

TODO:

- print messages only when debug mode is on
- config read in main() is lost

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #60
Closes #12
Closes #7
Closes #1
ligurio added a commit that referenced this issue Apr 20, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop" that replaces
file operation with no operation. More error injections are coming.

TODO:

- print messages only when debug mode is on
- config read in main() is lost
- make op_regexp

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #60
Closes #12
Closes #7
Closes #1
ligurio added a commit that referenced this issue Apr 24, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop" that replaces
file operation with no operation. More error injections are coming.

TODO:

- config read in main() is lost
- protect conf struct with mutex

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #60
Closes #12
Closes #7
Closes #1
ligurio added a commit that referenced this issue Apr 26, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop" that replaces
file operation with no operation. More error injections are coming.

TODO:

- config read in main() is lost

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #60
Closes #12
Closes #7
Closes #1
ligurio added a commit that referenced this issue Apr 29, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop" that replaces
file operation with no operation. More error injections are coming.

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #1
Closes #7
Closes #12
Closes #60
ligurio added a commit that referenced this issue Apr 29, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop" that replaces
file operation with no operation. More error injections are coming.

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #1
Closes #7
Closes #12
Closes #60
ligurio added a commit that referenced this issue Apr 29, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop" that replaces
file operation with no operation. More error injections are coming.

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #1
Closes #7
Closes #12
Closes #60
ligurio added a commit that referenced this issue Apr 29, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop" that replaces
file operation with no operation. More error injections are coming.

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #1
Closes #7
Closes #12
Closes #60
ligurio added a commit that referenced this issue Apr 29, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

TODO

- cannot read config when a single errinj section there

There is only one supported error injection - "errinj_noop" that replaces
file operation with no operation. More error injections are coming.

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #1
Closes #7
Closes #12
Closes #60
ligurio added a commit that referenced this issue Apr 30, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop" that replaces
file operation with no operation. More error injections are coming.

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #1
Closes #7
Closes #12
Closes #60
ligurio added a commit that referenced this issue Apr 30, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible was imported
third-party C library [1].

There is only one supported error injection - "errinj_noop" that replaces
file operation with no operation. More error injections are coming.

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #1
Closes #7
Closes #12
Closes #60
ligurio added a commit that referenced this issue Apr 30, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible third-party C
library [1] has been imported.

There is only one supported error injection - "errinj_noop" that replaces
file operation with no operation. More error injections are coming.

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #1
Closes #7
Closes #12
Closes #60
ligurio added a commit that referenced this issue Apr 30, 2021
Patch adds command line options and configuration file support
to unreliablefs.

unreliablefs can be managed in runtime using simple .INI configuration
file. unreliablefs config uses configuration language which provides a
structure similar to what's found in Microsoft Windows INI files or used
by configparser Python module [2]. To make it possible third-party C
library [1] has been imported.

There is only one supported error injection - "errinj_noop" that replaces
file operation with no operation. More error injections are coming.

1. https://github.com/benhoyt/inih
2. https://docs.python.org/3/library/configparser.html

Closes #3
Closes #18

Closes #1
Closes #7
Closes #12
Closes #60
ligurio added a commit that referenced this issue May 5, 2021
@ligurio
Copy link
Owner Author

ligurio commented May 5, 2021

3.1. Disk drives that do not honor sync requests
Unfortunately, most consumer-grade mass storage devices lie about syncing. Disk drives will report that content is safely on persistent media as soon as it reaches the track buffer and before actually being written to oxide. This makes the disk drives seem to operate faster (which is vitally important to the manufacturer so that they can show good benchmark numbers in trade magazines). And in fairness, the lie normally causes no harm, as long as there is no power loss or hard reset prior to the track buffer actually being written to oxide. But if a power loss or hard reset does occur, and if that results in content that was written after a sync reaching oxide while content written before the sync is still in a track buffer, then database corruption can occur.

USB flash memory sticks seem to be especially pernicious liars regarding sync requests. One can easily see this by committing a large transaction to an SQLite database on a USB memory stick. The COMMIT command will return relatively quickly, indicating that the memory stick has told the operating system and the operating system has told SQLite that all content is safely in persistent storage, and yet the LED on the end of the memory stick will continue flashing for several more seconds. Pulling out the memory stick while the LED is still flashing will frequently result in database corruption.

Note that SQLite must believe whatever the operating system and hardware tell it about the status of sync requests. There is no way for SQLite to detect that either is lying and that writes might be occurring out-of-order. However, SQLite in WAL mode is far more forgiving of out-of-order writes than in the default rollback journal modes. In WAL mode, the only time that a failed sync operation can cause database corruption is during a checkpoint operation. A sync failure during a COMMIT might result in loss of durability but not in a corrupt database file. Hence, one line of defense against database corruption due to failed sync operations is to use SQLite in WAL mode and to checkpoint as infrequently as possible.

https://www.sqlite.org/howtocorrupt.html

ligurio added a commit that referenced this issue May 10, 2021
ligurio added a commit that referenced this issue May 10, 2021
ligurio added a commit that referenced this issue Jun 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant