The actual number of cells in data section is not consistent with keyword '$TOT'

[vtosevski]

Dear Mike,

I have run into issue and couldn't find references to it elsewhere. I am working with a messy dataset with uncontrolled batch effects and a lot of non-cells per file. I decided to pre-gate those in FlowJo and export only "cells" gate for further processing and analysis. However, attempting to read in the newly created FCS files with flowCore gives me the error message

Error in FUN(X[[i]], ...) : 
  file Sample_017.fcs seems to be corrupted. 
 The actual number of cells in data section (49743) is not consistent with keyword '$TOT' (49812)

Have you seen this before and can suggest the way forward? Is there a way to make flowCore ignore the $TOT keyword and proceed (assuming I could trust the file itself)? I have tried ignore.text.offset argument but that didn't change anything.

Look forward to your feedback!

Vinko

[mikejiang]

The only way to bypass this check is to use which.lines to ask parser only select a subset of cells, in your case set which.lines = 49743. Note that it will slow down the reading.
Otherwise you will have to manually correct $TOT in FCS before using flowCore.

[vtosevski]

Thanks Mike! How do I learn the dimensions of the expression matrix without reading in the file first? I would gladly correct the $TOT but how do I find out the exact number to put there? Admittedly, I would need the same information in order to use which.lines argument. Vinko

…

On Wed, Apr 18, 2018, 23:16 Mike Jiang ***@***.***> wrote: The only way to bypass this check is to use which.lines to ask parser only select a subset of cells, in your case set which.lines = 49743. Note that it will slow down the reading. Otherwise you will have to manually correct $TOT in FCS before using flowCore. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#108 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AR3yjraBmfnMciQVFlkrQc4EneoQBXy8ks5tp60tgaJpZM4Tasg0> .

[mikejiang]

Maybe the right question is how can get flowJo export the correct FCS in the first place in stead of spending time on this mal-formated FCS. Otherwise you won't be confident about the analysis down the road,would you?
cc @jspidlen

[JosefFlowJo]

Thanks for cc-ing me Mike!

Vinko, FlowJo tells you the number of events in the "#Cells" column for every sample/population, and this is the number that should be in your $TOT keyword. Having said that, Mike is right, you definitely shouldn't need to adjust that manually. Generally, when you export a sub-population from FlowJo, we will place the correct number of events in the $TOT keyword. I just did a quick sanity check (see screenshot below), and it seems to be working as expected. So I suspect that something else may be going on in your case. Would you be able to share maybe an ACS file export for me to look at? Also, which version of FlowJo are you using? You can also email me directly at josefs@flowjo.com and I'd be more than happy to help you troubleshoot this.

--
Josef Spidlen, Ph.D. | josefs@flowjo.com
Director of Bioinformatics | FlowJo, LLC
385 Williamson Way
Ashland, OR 97520 USA

[vtosevski]

@mikejiang I absolutely share your sentiment about this and I was hesitant posting this in the first place as this clearly is not the flowCore issue. But I was hoping there might be a way for flowCore to account for this and allow me to move forward. I know this is not fair in a way, asking you if you could help clean someone else's mess, but felt was worth a try...

Good idea to include Josef, I will reach out to him directly.

Thanks!

[nlaniewski]

@mikejiang I'm having a similar - somewhat inverse - issue @vtosevski with the $TOT keyword and write.FCS function; newly written, subsampled FCS files then loaded into either FlowJo or FCS Express 6 do not contain the appropriate $TOT value.

data(GvHD)

tmp <- GvHD[[1]]
exprs(tmp) <- exprs(tmp)[sample(1:nrow(exprs(tmp)), 1000), ]

suppressWarnings(write.FCS(GvHD[[1]],"GvHD1.fcs"))
suppressWarnings(write.FCS(tmp,"GvHD1_sub.fcs"))

Reading these files back into flowCore and extracting the $TOT value seems to be correct; loading the subsampled file into either of the aforementioned software packages is problematic as the $TOT value is retained from the original sample (GvHD[[1]]@description$`$TOT = 3420).

FCS Express 6 generates this error when trying to read/parse the header from the flowCore subsampled FCS (1000 total events, as set in exprs(tmp)[sample(1:nrow(exprs(tmp)), 1000), ]):

The data start and end do not match:
DataStart=2399,
DataEnd=34398,
DataBlockWidth=32000,
RowCount=3420,
BytesPerRow=32,
ExpectedDataBlockSizeInBytes=109440,
Difference From Expected=77440

This behavior is observed using R 3.5.1 and flowCore 1.46.1.

Using R 3.4.0 and flowCore 1.44.2 (before I updated and encountered this issue) all writing and reading of FCS files behaved as expected. $TOT in both flowCore and software contained the appropriate values.

Thanks for your time! Please let me know if I've overlooked anything that might be causing this. I've since rolled back to using the older versions of R and flowCore but ideally would like to use your updated/current flowCore package.

[mikejiang]

It was the duplicated $TOT with the old value interferes the FCS Express. Pull the latest trunk and give it try. Let me know if it works.

[nlaniewski]

@mikejiang Excellent! Thanks for the quick response. It's working for me now using flowCore_1.47.4; files created using write.FCS display the correct $TOT in software (FlowJo/FCS Express).

Not sure if this fixes @vtosevski issue...

[gfinak]

@vtosevski is this issue resolved for you?

[vtosevski]

Yes, it's solved. As Mike initially suspected and everyone agreed it made sense, the issue was with FlowJo not updating gate statistics properly before exporting the gates. The flowcore simply identified the problem, didn't cause it. Updating gate stats manually in FlowJo before exporting was sufficient to export a "proper" fcs file and, expectedly, flowcore had no issues working with those. Thanks! Vinko

…

On Fri, Aug 17, 2018, 19:43 Greg Finak ***@***.***> wrote: @vtosevski <https://github.com/vtosevski> is this issue resolved for you? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#108 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AR3yjkV_m6k5BIw5-7enIhd1MpOz_nBiks5uRwDfgaJpZM4Tasg0> .

[gfinak]

Thanks @vtosevski

[jakesauter]

This worked for me:

sample <- ### Your flowFrame of interest

new_sample <-
flowCore::flowFrame(
exprs(sample)
)

keyword(new_sample)["$TOT"] <- nrow(new_sample)
keyword(new_sample)["$TIMESTEP"] <- keyword(sample)["$TIMESTEP"]

new_sample_fcs_file <-
paste0(sample_name, ".fcs")

write.FCS(
new_sample,
new_sample_fcs_file
)

flow_auto_qc(
new_sample_fcs_file
)

file.remove(
new_sample_fcs_file
)

[Shuaiwei-Wang]

Yes, it's solved. As Mike initially suspected and everyone agreed it made sense, the issue was with FlowJo not updating gate statistics properly before exporting the gates. The flowcore simply identified the problem, didn't cause it. Updating gate stats manually in FlowJo before exporting was sufficient to export a "proper" fcs file and, expectedly, flowcore had no issues working with those. Thanks! Vinko
…
On Fri, Aug 17, 2018, 19:43 Greg Finak @.***> wrote: @vtosevski https://github.com/vtosevski is this issue resolved for you? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#108 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AR3yjkV_m6k5BIw5-7enIhd1MpOz_nBiks5uRwDfgaJpZM4Tasg0 .

Yes, it's solved. As Mike initially suspected and everyone agreed it made sense, the issue was with FlowJo not updating gate statistics properly before exporting the gates. The flowcore simply identified the problem, didn't cause it. Updating gate stats manually in FlowJo before exporting was sufficient to export a "proper" fcs file and, expectedly, flowcore had no issues working with those. Thanks! Vinko
…
On Fri, Aug 17, 2018, 19:43 Greg Finak @.***> wrote: @vtosevski https://github.com/vtosevski is this issue resolved for you? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#108 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AR3yjkV_m6k5BIw5-7enIhd1MpOz_nBiks5uRwDfgaJpZM4Tasg0 .

Hi @vtosevski Could you please share some information about how manually change $TOT?

[SamGG]

@Shuaiwei-Wang Read the discussion, there is nothing as "manually change $TOT". Wait for FJ finishing updating stuff before exporting.