Conversation
port8080
force-pushed
the
ajit/add-xet-telemetry
branch
from
0f05b62 to
b08974a
Compare
Get the telemetry data to the py layer. This will be sent over the telemetry endpoint later. Related to STO-7
port8080
force-pushed
the
ajit/add-xet-telemetry
branch
from
b08974a to
bd328fd
Compare
| pub trait ReconstructionClient { | ||
| /// Get a entire file by file hash. | ||
| async fn get_file(&self, hash: &MerkleHash, writer: &mut Box<dyn Write + Send>) -> Result<()>; | ||
| async fn get_file(&self, hash: &MerkleHash, writer: &mut Box<dyn Write + Send>) -> Result<u64>; |
There was a problem hiding this comment.
add a comment for what this return value is, presumably amount written to writer
| } | ||
| Ok(total_len as usize) | ||
| // Todo: return the bytes which were read from the cache for telemetry | ||
| Ok(total_len as u64) |
There was a problem hiding this comment.
this total len is the uncompressed length of the section, not the number of bytes over the network which is what I think you're attempting to track that number should be the difference in the url_range end - start for each piece.
| cas_data.pending_file_info.clear(); | ||
|
|
||
| Ok(cas_hash) | ||
| Ok((cas_hash, compressed_bytes_len as u64)) |
There was a problem hiding this comment.
nit: using compressed_bytes_len from client side is unreliable, people can inject any number here and that does not impact clean process correctness.
| let mut new_cas_data = take(cas_data_accumulator.deref_mut()); | ||
| drop(cas_data_accumulator); // Release the lock. | ||
| register_new_cas_block( | ||
| let (_cas_hash, compressed_bytes) = register_new_cas_block( |
There was a problem hiding this comment.
This compressed_bytes
- doesn't include all bytes produced by this file after dedup: they are also produced in
fn dedup(...). I noticed you modified that function but the return value is not used? - data in this
cas_data_accumulatormay also include bytes produced by other files cleaned before this file using the same PointerFileTranslator.
seanses
left a comment
seanses
left a comment
There was a problem hiding this comment.
The logic to count bytes produced by one file after dedup doesn't seem correct.
|
Lots of things have changed since the time of this PR, closing it out. If we choose to add telemetry later on we will likely have to start from scratch anyway. |
4 participants
Get the telemetry data to the py layer. This will be sent over the telemetry endpoint later.
Related to STO-7