[BUG] inconsistent state may occur

[jxlpzqc]

Hi, we deploy panamax on our mirror site, and find a crate package exists in crates-index, but misses in crates files.

panamax/src/crates.rs

Lines 295 to 311 in 2024b5b

	Err(e) => {
	eprintln!("Downloading failed: {e:?}");
	}
	}
	}
	// Delete any removed crates
	for rc in removed_crates {
	// Try to remove the file, but ignore it if it doesn't exist
	let _ = fs::remove_file(repo_path.join(rc));
	}
	// Set master to origin/master.
	//
	// Note that this means config.json changes will have to be rewritten on every sync.
	fast_forward(&repo_path)?;

In above code block, the program ignore DownloadError and fast-forward the git repo.

We suppose downloading of one crate meets network error. Next time, this failed crate is absolutely not in diff result of origin/master and master because fast-forward has been done, which leads to this version of crates will never get downloaded since this time.

Even worse, when syncing logs are cleared periodically, it is not easy to find which states have been broken. (the situation we meet.)

We suggest when error occurs, just stop going on, and wait for syncing next time to avoid inconsistent state occurs.

[rust1248]

Couldn't you use panamax verify?

[jxlpzqc]

Sure, panamax verify may address this issue to some extent, but when deployed in an automatic situation, how often should verifing runs?

Anyhow, the inconsistent states has been entered when download failed. Why not just cancel this broken job?

[k3d3]

The reason Panamax works the way it currently does is because when I first made the program, there were several crates which would repeatedly return a 403 error, and would break every single attempt to mirror the repository. That's why it logs rather than fails, because originally it would have failed every time.

@rust1248 is right; this is the use case for the verify command. The idea is that depending on how averse you are to having an incomplete mirror, you can either run verify after each sync, or on a regular interval like once a week, or at its lowest, only when a mirror user complains that a package is missing.

With that said, I do feel like the logging story for Panamax is lacking, and doesn't lend the best interface for automation right now. I need to work on that.

[jxlpzqc]

@k3d3 Thank you very much for giving such a detailed answer. We are also so grateful to panamax due to its huge role in our mirror site. I have fully understood the reason for this design, and will close this issue.