Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(ci): Increase the full sync timeout #5129

Merged
merged 3 commits into from Sep 13, 2022
Merged

fix(ci): Increase the full sync timeout #5129

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0efbb5e
Increase the full sync timeout to 28 hours
teor2345 Sep 11, 2022
b912b72
Update developer documentation for the new constant location
teor2345 Sep 11, 2022
293181f
Merge branch 'main' into up-full-sync-timeout
mergify[bot] Sep 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 5 additions & 5 deletions book/src/dev/continuous-integration.md
View file
Open in desktop
Expand Up @@ -47,7 +47,7 @@ Please shut down large instances when they are not being used.
### Automated Deletion

The [Delete GCP Resources](https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/delete-gcp-resources.yml)
workflow automatically deletes instance templates, disks, and images older than 1 week.
workflow automatically deletes instance templates, disks, and images older than a few days.

Running instances and their disks are protected from deletion.

Expand Down Expand Up @@ -91,7 +91,7 @@ https://github.com/ZcashFoundation/zebra/runs/8181760421?check_suite_focus=true#
The earlier failure can also be in another job, check out the whole workflow run for details.
(Use the "Summary" button on the top left of the job details, and zoom in.)

### Resolving CI Sync Timeouts
### Fixing CI Sync Timeouts

CI sync jobs near the tip will take different amounts of time as:
- the blockchain grows, and
Expand All @@ -111,11 +111,11 @@ To fix a CI sync timeout, follow these steps until the timeouts are fixed:

5. If a Rust test fails with "command did not log any matches for the given regex, within the ... timeout":

a. If it's the full sync test, [increase the full sync timeout](https://github.com/ZcashFoundation/zebra/commit/9fb87425b76ba3747985ea2f22043ff0276a03bd#diff-8fbc73b0a92a4f48656ffe7d85d55c612c755202dcb7284d8f6742a38a6e9614R367)
a. If it's the full sync test, [increase the full sync timeout](https://github.com/ZcashFoundation/zebra/pull/5129/files)

b. If it's an update sync test, [increase the update sync timeouts](https://github.com/ZcashFoundation/zebra/commit/9fb87425b76ba3747985ea2f22043ff0276a03bd#diff-92f93c26e696014d82c3dc1dbf385c669aa61aa292f44848f52167ab747cb6f6R51)

### Resolving Duplicate Dependencies in `Check deny.toml bans`
### Fixing Duplicate Dependencies in `Check deny.toml bans`

Zebra's CI checks for duplicate crate dependencies: multiple dependencies on different versions of the same crate.
If a developer or dependabot adds a duplicate dependency, the `Check deny.toml bans` CI job will fail.
Expand Down Expand Up @@ -149,7 +149,7 @@ To fix duplicate dependencies, follow these steps until the duplicate dependenci

4. Repeat step 3 until the dependency warnings are fixed. Adding a single exception can resolve multiple warnings.

### Resolving Disk Full Errors
### Fixing Disk Full Errors

If the Docker cached state disks are full, increase the disk sizes in:
- [deploy-gcp-tests.yml](https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/deploy-gcp-tests.yml)
Expand Down
9 changes: 5 additions & 4 deletions zebrad/tests/common/sync.rs
View file
Open in desktop
Expand Up @@ -74,6 +74,10 @@ pub const LARGE_CHECKPOINT_TIMEOUT: Duration = Duration::from_secs(180);
/// cause the test to fail.
pub const FINISH_PARTIAL_SYNC_TIMEOUT: Duration = Duration::from_secs(11 * 60 * 60);

/// The maximum time to wait for Zebrad to synchronize up to the chain tip starting from the
/// genesis block.
pub const FINISH_FULL_SYNC_TIMEOUT: Duration = Duration::from_secs(28 * 60 * 60);

/// The test sync height where we switch to using the default lookahead limit.
///
/// Most tests only download a few blocks. So tests default to the minimum lookahead limit,
Expand Down Expand Up @@ -366,9 +370,6 @@ pub fn create_cached_database_height(
) -> Result<()> {
eprintln!("creating cached database");

// 24 hours
let timeout = Duration::from_secs(24 * 60 * 60);

// Use a persistent state, so we can handle large syncs
let mut config = cached_mandatory_checkpoint_test_config()?;
// TODO: add convenience methods?
Expand All @@ -381,7 +382,7 @@ pub fn create_cached_database_height(
let mut child = dir
.with_exact_config(&config)?
.spawn_child(args!["start"])?
.with_timeout(timeout)
.with_timeout(FINISH_FULL_SYNC_TIMEOUT)
.bypass_test_capture(true);

let network = format!("network: {},", network);
Expand Down