Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use fsync to flush metadata when reoperning a file with new headers #308

Merged
merged 1 commit into from Apr 28, 2021
Merged

use fsync to flush metadata when reoperning a file with new headers #308

merged 1 commit into from Apr 28, 2021

Conversation

samoht
Copy link
Member

@samoht samoht commented Apr 27, 2021 •
edited

Not totally sure about this one: this will probably slow down single writer/multiple instances within the same process (but probably not much as this is not a very common operation) ; but that'll speed up signalling of when a merge is done for other processes opened in read-only mode.

@samoht
Copy link
Member Author

samoht commented Apr 28, 2021

Pushed an update. fsync is now called after merges to make sure read-only instance are updated correctly. Will merge once the CI is green.

@samoht samoht merged commit dcce632 into mirage:master Apr 28, 2021
@samoht samoht deleted the fsync branch April 28, 2021 20:57
@samoht samoht mentioned this pull request Apr 29, 2021
Closed
samoht added a commit to samoht/opam-repository that referenced this pull request Apr 29, 2021
@samoht
[new release] index and index-bench (1.3.1)
cc6a189 
CHANGES:

## Fixed

- Reduce allocations during merge (mirage/index#274, mirage/index#277)

- Protect concurrent syncs with a lock (mirage/index#309)

- Fixed a performance issue for `Index.sync` when there is a blocking merge in
  progress: the `log_async` file was not cached properly and fully reloaded
  from disk every time. (mirage/index#310)

- Release the merge lock if a merge raises an exception (mirage/index#312)

- Added fsync after `Index.clear` to signal more quickly to read-only instances
  than something has changed in the file (mirage/index#308)

## Changed

- Specialise `IO.v` to create read-only or read-write instances. (mirage/index#291)

- `clear` removes the files on disks and opens new ones containing only the
  header. (mirage/index#288, mirage/index#307, mirage/index#317)
samoht added a commit to samoht/opam-repository that referenced this pull request Apr 29, 2021
@samoht
[new release] index and index-bench (1.3.1)
c54b951 
CHANGES:

## Fixed

- Reduce allocations during merge (mirage/index#274, mirage/index#277)

- Protect concurrent syncs with a lock (mirage/index#309)

- Fixed a performance issue for `Index.sync` when there is a blocking merge in
  progress: the `log_async` file was not cached properly and fully reloaded
  from disk every time. (mirage/index#310)

- Release the merge lock if a merge raises an exception (mirage/index#312)

- Added fsync after `Index.clear` to signal more quickly to read-only instances
  than something has changed in the file (mirage/index#308)

## Changed

- Specialise `IO.v` to create read-only or read-write instances. (mirage/index#291)

- `clear` removes the files on disks and opens new ones containing only the
  header. (mirage/index#288, mirage/index#307, mirage/index#317)
@samoht samoht mentioned this pull request Apr 29, 2021
Merged
samoht added a commit to samoht/opam-repository that referenced this pull request Apr 29, 2021
@samoht
[new release] index and index-bench (1.3.1)
a323a1e 
CHANGES:

## Fixed

- Reduce allocations during merge (mirage/index#274, mirage/index#277)

- Protect concurrent syncs with a lock (mirage/index#309)

- Fixed a performance issue for `Index.sync` when there is a blocking merge in
  progress: the `log_async` file was not cached properly and fully reloaded
  from disk every time. (mirage/index#310)

- Release the merge lock if a merge raises an exception (mirage/index#312)

- Added fsync after `Index.clear` to signal more quickly to read-only instances
  than something has changed in the file (mirage/index#308)

## Changed

- Specialise `IO.v` to create read-only or read-write instances. (mirage/index#291)

- `clear` removes the files on disks and opens new ones containing only the
  header. (mirage/index#288, mirage/index#307, mirage/index#317)
samoht added a commit to samoht/opam-repository that referenced this pull request Apr 29, 2021
@samoht
[new release] index (1.3.1)
16ff00f 
CHANGES:

- Reduce allocations during merge (mirage/index#274, mirage/index#277)

- Protect concurrent syncs with a lock (mirage/index#309)

- Fixed a performance issue for `Index.sync` when there is a blocking merge in
  progress: the `log_async` file was not cached properly and fully reloaded
  from disk every time. (mirage/index#310)

- Release the merge lock if a merge raises an exception (mirage/index#312)

- Added fsync after `Index.clear` to signal more quickly to read-only instances
  than something has changed in the file (mirage/index#308)

- Specialise `IO.v` to create read-only or read-write instances. (mirage/index#291)

- `clear` removes the files on disks and opens new ones containing only the
  header. (mirage/index#288, mirage/index#307, mirage/index#317)
craigfe added a commit to craigfe/opam-repository that referenced this pull request Jun 16, 2021
@craigfe
[new release] index-bench and index (1.4.0)
91bbc11 
CHANGES:

## Fixed

- Fixed a crash-consistency bug due to a potential flush of an incomplete entry
  to disk. Entries are now flushed as complete strings. (mirage/index#301)

- Fixed a performance issue for `Index.sync` when there is a blocking merge in
  progress: the `log_async` file was not cached properly and fully reloaded
  from disk every time. (mirage/index#310)

- Added fsync after `Index.clear` to signal more quickly to read-only instances
  than something has changed in the file (mirage/index#308)

- Attempt to recover from `log_async` invariant violations during an explicit
  sync operation, rather than failing immediately. (mirage/index#329)

## Changed

- Release overly defensive warnings occuring when pre-fetching the disk. (mirage/index#322)

- Specialise `IO.v` to create read-only or read-write instances. (mirage/index#291)

- Optimised the in-memory representation of index handles and intermediate
  buffers, resulting in a significant reduction in memory use. (mirage/index#273, mirage/index#279)

- Benches are now executed 3 times and a new option `nb-exec` has been added (mirage/index#292)

- `clear` removes the files on disks and opens new ones containing only the
  header. (mirage/index#288)

- `Index.Make` now requires an implementation of a monotonic time source.
  (mirage/index#321)

- The `Index.Make` functor now takes a single `Platform` argument containing
  all system dependencies (i.e. `IO`, `Clock`, `Semaphore` and `Thread`). The
  `Platform` module holds the necessary types for these modules. (mirage/index#321, mirage/index#330)

## Added

- Added benchmarks that replay a trace of index operations. (mirage/index#300)

- Log reporter for the benches
@craigfe craigfe mentioned this pull request Jun 16, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@samoht