0 byte dat file

[stevendanna]

We recently had a user report that a Habitat supervisor was failing to start. The supervisor log showed:

Aug 15 18:30:17 redacted.host hab[1242]: hab-sup(ER)[components/sup/src/error.rs:449:9]: Butterfly error: Error reading or writing to DatFile, /hab/sup/default/data/cab66ba4cf12423db7b6cce2a1d76c54.rst, failed to fill whole buffer

Further investigation showed that /hab/sup/default/data/cab66ba4cf12423db7b6cce2a1d76c54.rst existed but was a 0-byte file.

While we haven't been able to investigate the problem further yet, the code that handles the writes:

habitat/components/butterfly/src/rumor/dat_file.rs

Lines 234 to 267 in 52b8c4e

	pub fn write(&self, server: &Server) -> Result<usize> {
	let mut header = Header::default();
	let tmp_path = self
	.path
	.with_extension(thread_rng().gen_ascii_chars().take(8).collect::<String>());
	{
	let file = OpenOptions::new()
	.create(true)
	.write(true)
	.truncate(true)
	.open(&tmp_path)
	.map_err(|err| Error::DatFileIO(tmp_path.clone(), err))?;
	let mut writer = BufWriter::new(file);
	self.init(&mut writer)?;
	header.member_len = self.write_member_list(&mut writer, &server.member_list)?;
	header.service_len = self.write_rumor_store(&mut writer, &server.service_store)?;
	header.service_config_len =
	self.write_rumor_store(&mut writer, &server.service_config_store)?;
	header.service_file_len =
	self.write_rumor_store(&mut writer, &server.service_file_store)?;
	header.election_len = self.write_rumor_store(&mut writer, &server.election_store)?;
	header.update_len = self.write_rumor_store(&mut writer, &server.update_store)?;
	header.departure_len = self.write_rumor_store(&mut writer, &server.departure_store)?;
	writer
	.seek(SeekFrom::Start(1))
	.map_err(|err| Error::DatFileIO(self.path.clone(), err))?;
	self.write_header(&mut writer, &header)?;
	writer
	.flush()
	.map_err(|err| Error::DatFileIO(self.path.clone(), err))?;
	}
	fs::rename(&tmp_path, &self.path).map_err(|err| Error::DatFileIO(self.path.clone(), err))?;
	Ok(0)
	}

appears to be missing at least two calls to fsync if we want those writes to be durable. We likely need an fsync on the temporary file before the rename to ensure that the content of the file has been written to disk and an fsync on the enclosing directory after the rename to ensure that the rename operation is flushed to disk.

[mwrock]

Closed by #5461