Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Traverse gced pack stores #1919

Merged
merged 1 commit into from
Oct 6, 2022
Merged

Traverse gced pack stores #1919

merged 1 commit into from
Oct 6, 2022

Conversation

icristescu
Copy link
Contributor

The traverse_pack_files on v2 stores are anyway broken #1903. But even more so on gced stores: the traversal of the suffix file needs to read gced objects, without knowing their size.

We will need to come back to this once we have the good gc.

@icristescu icristescu changed the base branch from blocking-dummy-gc to main September 26, 2022 08:52
@icristescu icristescu changed the title WIP: Traverse gced pack stores Traverse gced pack stores Sep 26, 2022
@icristescu icristescu marked this pull request as ready for review September 26, 2022 09:00
@codecov-commenter
Copy link

codecov-commenter commented Sep 26, 2022 •
edited
Loading

Codecov Report

Merging #1919 (e877163) into main (36ad14b) will increase coverage by 0.06%.
The diff coverage is 28.57%.

❗ Current head e877163 differs from pull request most recent head 18e14b6. Consider uploading reports for the commit 18e14b6 to get more accurate results

@@            Coverage Diff             @@
##             main    #1919      +/-   ##
==========================================
+ Coverage   64.61%   64.67%   +0.06%     
==========================================
  Files         132      132              
  Lines       15693    15702       +9     
==========================================
+ Hits        10140    10156      +16     
+ Misses       5553     5546       -7
Impacted Files Coverage Δ
src/irmin-pack/unix/traverse_pack_file.ml 64.13% <28.57%> (+7.52%) ⬆️
src/irmin-test/store.ml 94.89% <0.00%> (-0.06%) ⬇️
src/irmin-pack/pack_value.ml 82.41% <0.00%> (+1.09%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@icristescu icristescu changed the title Traverse gced pack stores (wip) Traverse gced pack stores Sep 26, 2022
@icristescu icristescu force-pushed the traverse branch 2 times, most recently from 5b5b93f to 296af80 Compare September 28, 2022 08:54
@icristescu icristescu changed the title (wip) Traverse gced pack stores Traverse gced pack stores Sep 28, 2022
@icristescu
Copy link
Contributor Author

This is ready for review. It fixes the traverse pack functions in the tezos-node cli to work on gced stores.

@icristescu icristescu mentioned this pull request Oct 4, 2022
Closed
34 tasks
metanivek
metanivek reviewed Oct 4, 2022
View reviewed changes
Copy link
Member

@metanivek metanivek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I'm not sure where/how this code is used but LGTM. Left a couple of comments for small improvements.

src/irmin-pack/unix/traverse_pack_file.ml Outdated Show resolved Hide resolved
src/irmin-pack/unix/traverse_pack_file.ml Outdated Show resolved Hide resolved
@icristescu icristescu merged commit ed988e9 into mirage:main Oct 6, 2022
@icristescu icristescu deleted the traverse branch October 6, 2022 08:56
@metanivek metanivek mentioned this pull request Oct 6, 2022
Merged
metanivek added a commit to metanivek/opam-repository that referenced this pull request Oct 6, 2022
@metanivek
[new release] ppx_irmin, libirmin, irmin, irmin-tezos, irmin-tezos-ut…
029f725 
…ils, irmin-test, irmin-pack, irmin-mirage, irmin-mirage-graphql, irmin-mirage-git, irmin-http, irmin-graphql, irmin-git, irmin-fs, irmin-containers, irmin-cli, irmin-chunk and irmin-bench (3.4.2)

CHANGES:

### Added

- **irmin**
  - Add `test_set_and_get*` functions to retrieve commit associated with an update to the store (mirage/irmin#2075, @patricoferris)

- **irmin-graphql**
  -  Expose `test_set_and_get` function as a new mutation (mirage/irmin#2075, @patricoferris)
  -  Add `contents_hash` function to get a value's hash (mirage/irmin#2099, @patricoferris)

- **irmin-pack**
  - Expose `Gc.cancel` to abort a running GC (mirage/irmin#2101, @art-w)

- **irmin-tezos-utils**
  - Add package `irmin-tezos-utils` containing a graphical tool for manual pack
    files analysis. (mirage/irmin#1939, @clecat)

### Changed

- **irmin-pack**
  - `irmin_pack_mem` no longer exposes disk specifics functions (mirage/irmin#2081,
  @icristescu)
  - Move unix specific details for `Pack_key` and `Pack_value` from `irmin-pack`
    to `irmin-pack.unix` (mirage/irmin#2084, @metanivek)
  - Remove unnecessary files at `open_rw` and after a failed GC (mirage/irmin#2095, @art-w)

### Fixed

- **irmin-pack**
  - Fix data race in RO instances when reading control file (mirage/irmin#2100, @Ngoguey42)
  - Fix bugs in gc related to commits that share the same tree. (mirage/irmin#2106,
    @icristescu)

### Fixed

- **irmin-pack**
  - Fix the traverse pack files commands in the `irmin-tezos` CLI to work with
    gced stores. (mirage/irmin#1919, @icristescu)
metanivek added a commit to metanivek/opam-repository that referenced this pull request Oct 6, 2022
@metanivek
[new release] ppx_irmin, libirmin, irmin, irmin-tezos, irmin-tezos-ut…
ff7795a 
…ils, irmin-test, irmin-pack, irmin-mirage, irmin-mirage-graphql, irmin-mirage-git, irmin-http, irmin-graphql, irmin-git, irmin-fs, irmin-containers, irmin-cli, irmin-chunk and irmin-bench (3.4.2)

CHANGES:

### Added

- **irmin**
  - Add `test_set_and_get*` functions to retrieve commit associated with an update to the store (mirage/irmin#2075, @patricoferris)

- **irmin-graphql**
  -  Expose `test_set_and_get` function as a new mutation (mirage/irmin#2075, @patricoferris)
  -  Add `contents_hash` function to get a value's hash (mirage/irmin#2099, @patricoferris)

- **irmin-pack**
  - Expose `Gc.cancel` to abort a running GC (mirage/irmin#2101, @art-w)

- **irmin-tezos-utils**
  - Add package `irmin-tezos-utils` containing a graphical tool for manual pack
    files analysis. (mirage/irmin#1939, @clecat)

### Changed

- **irmin-pack**
  - `irmin_pack_mem` no longer exposes disk specifics functions (mirage/irmin#2081,
  @icristescu)
  - Move unix specific details for `Pack_key` and `Pack_value` from `irmin-pack`
    to `irmin-pack.unix` (mirage/irmin#2084, @metanivek)
  - Remove unnecessary files at `open_rw` and after a failed GC (mirage/irmin#2095, @art-w)

### Fixed

- **irmin-pack**
  - Fix data race in RO instances when reading control file (mirage/irmin#2100, @Ngoguey42)
  - Fix bugs in gc related to commits that share the same tree. (mirage/irmin#2106,
    @icristescu)
  - Fix the traverse pack files commands in the `irmin-tezos` CLI to work with
    gced stores. (mirage/irmin#1919, @icristescu)
metanivek added a commit to metanivek/opam-repository that referenced this pull request Oct 6, 2022
@metanivek
[new release] ppx_irmin, libirmin, irmin, irmin-tezos, irmin-tezos-ut…
fe1a48a 
…ils, irmin-test, irmin-pack, irmin-mirage, irmin-mirage-graphql, irmin-mirage-git, irmin-http, irmin-graphql, irmin-git, irmin-fs, irmin-containers, irmin-cli, irmin-chunk and irmin-bench (3.4.2)

CHANGES:

### Added

- **irmin**
  - Add `test_set_and_get*` functions to retrieve commit associated with an update to the store (mirage/irmin#2075, @patricoferris)

- **irmin-graphql**
  -  Expose `test_set_and_get` function as a new mutation (mirage/irmin#2075, @patricoferris)
  -  Add `contents_hash` function to get a value's hash (mirage/irmin#2099, @patricoferris)

- **irmin-pack**
  - Expose `Gc.cancel` to abort a running GC (mirage/irmin#2101, @art-w)

- **irmin-tezos-utils**
  - Add package `irmin-tezos-utils` containing a graphical tool for manual pack
    files analysis. (mirage/irmin#1939, @clecat)

### Changed

- **irmin-pack**
  - `irmin_pack_mem` no longer exposes disk specifics functions (mirage/irmin#2081,
  @icristescu)
  - Move unix specific details for `Pack_key` and `Pack_value` from `irmin-pack`
    to `irmin-pack.unix` (mirage/irmin#2084, @metanivek)
  - Remove unnecessary files at `open_rw` and after a failed GC (mirage/irmin#2095, @art-w)

### Fixed

- **irmin-pack**
  - Fix data race in RO instances when reading control file (mirage/irmin#2100, @Ngoguey42)
  - Fix bugs in gc related to commits that share the same tree. (mirage/irmin#2106,
    @icristescu)
  - Fix the traverse pack files commands in the `irmin-tezos` CLI to work with
    gced stores. (mirage/irmin#1919, @icristescu)
icristescu pushed a commit to metanivek/opam-repository that referenced this pull request Oct 7, 2022
[new release] ppx_irmin, libirmin, irmin, irmin-tezos, irmin-tezos-ut…
8d6af4e 
…ils, irmin-test, irmin-pack, irmin-mirage, irmin-mirage-graphql, irmin-mirage-git, irmin-http, irmin-graphql, irmin-git, irmin-fs, irmin-containers, irmin-cli, irmin-chunk and irmin-bench (3.4.2)

CHANGES:

### Added

- **irmin**
  - Add `test_set_and_get*` functions to retrieve commit associated with an update to the store (mirage/irmin#2075, @patricoferris)

- **irmin-graphql**
  -  Expose `test_set_and_get` function as a new mutation (mirage/irmin#2075, @patricoferris)
  -  Add `contents_hash` function to get a value's hash (mirage/irmin#2099, @patricoferris)

- **irmin-pack**
  - Expose `Gc.cancel` to abort a running GC (mirage/irmin#2101, @art-w)

- **irmin-tezos-utils**
  - Add package `irmin-tezos-utils` containing a graphical tool for manual pack
    files analysis. (mirage/irmin#1939, @clecat)

### Changed

- **irmin-pack**
  - `irmin_pack_mem` no longer exposes disk specifics functions (mirage/irmin#2081,
  @icristescu)
  - Move unix specific details for `Pack_key` and `Pack_value` from `irmin-pack`
    to `irmin-pack.unix` (mirage/irmin#2084, @metanivek)
  - Remove unnecessary files at `open_rw` and after a failed GC (mirage/irmin#2095, @art-w)

### Fixed

- **irmin-pack**
  - Fix data race in RO instances when reading control file (mirage/irmin#2100, @Ngoguey42)
  - Fix bugs in gc related to commits that share the same tree. (mirage/irmin#2106,
    @icristescu)
  - Fix the traverse pack files commands in the `irmin-tezos` CLI to work with
    gced stores. (mirage/irmin#1919, @icristescu)
@irmaTS irmaTS added the tezos-support Support for bugs related to Tezos label Feb 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@metanivek metanivek metanivek approved these changes

Assignees
No one assigned
Labels
tezos-support Support for bugs related to Tezos
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@icristescu @codecov-commenter @metanivek @irmaTS