Quad gates from flowjo are converted to giant rectangles in Cytobank

[cciccole]

After import into Cytobank, quad gates from flowjo are not quad gates but instead are 4 rectangular gates. The gates are huge and their edges extend very far off axis. The statistics labels are also very far off the plot and thus not visible. In order to see the labels the gate must be dragged and resized many times, which is painful.

This issue affects other quad types in FJ such as curly quad and quad-disconnected-center. It doesn't seem to affect spider quad as badly (for this quad type, the edges extend only marginally off axis).

Should be fairly simple to adjust CytoML to not make the rectangles so large. They just need to slightly clear the scale maximum. This ensures that the gates capture all data points beyond the scale maximum, since these data points are binned at the scale maximum in Cytobank. This strategy may solve the problem with the labels without any label-specific logic, since I believe label position is set automatically based on general gate characteristics.

[mikejiang]

FJ is affected by this issue as well #20 , I am in the process of efforts of restoring quadgate

[mikejiang]

@cciccole , I think I need your help to figure out the correct format specification for QuadrantGate,
I am getting the error from cytobank

I think it may have to do with the fact that I didn't provide the json version of quadgate definition. But I thought it is not required (as I didn't need to do so for other gate types).
If it is indeed mandated for QuadGate, then what's the minimum required information. (I tried to look at the one output from cytobank, but it wasn't all clear too me, e.g. what is 'angle' field?)
here is the node I generated through CytoML

<gating:QuadrantGate id="Gate_7_Q0Q0IHN1Yg..">
  <gating:divider transformation-ref="Tr_Arcsinh_Comp_B710-A" compensation-ref="Spill_defaultCompensation" id="Gate_7_Q0Q0IHN1Yg..divider_1">
   <data-type:fcs-dimension name="Comp_G560-A"/>
   <gating:value>2.23594047371882</gating:value>
  </gating:divider>
  <gating:divider transformation-ref="Tr_Arcsinh_Comp_B710-A" compensation-ref="Spill_defaultCompensation" id="Gate_7_Q0Q0IHN1Yg..divider_2">
   <data-type:fcs-dimension name="Comp_G780-A"/>
   <gating:value>3.41168125519337</gating:value>
  </gating:divider>
  <gating:Quadrant id="Gate_9_Q0NSNysgNDVSQSs.">
   <gating:position divider_ref="Gate_7_Q0Q0IHN1Yg..divider_1" location="3.23594047371882"/>
   <gating:position divider_ref="Gate_7_Q0Q0IHN1Yg..divider_2" location="4.41168125519337"/>
  </gating:Quadrant>
  <gating:Quadrant id="Gate_10_Q0NSNysgNDVSQS0.">
   <gating:position divider_ref="Gate_7_Q0Q0IHN1Yg..divider_1" location="3.23594047371882"/>
   <gating:position divider_ref="Gate_7_Q0Q0IHN1Yg..divider_2" location="2.41168125519337"/>
  </gating:Quadrant>
  <gating:Quadrant id="Gate_8_Q0NSNy0gNDVSQSs.">
   <gating:position divider_ref="Gate_7_Q0Q0IHN1Yg..divider_1" location="1.23594047371882"/>
   <gating:position divider_ref="Gate_7_Q0Q0IHN1Yg..divider_2" location="4.41168125519337"/>
  </gating:Quadrant>
  <gating:Quadrant id="Gate_11_Q0NSNy0gNDVSQS0.">
   <gating:position divider_ref="Gate_7_Q0Q0IHN1Yg..divider_1" location="1.23594047371882"/>
   <gating:position divider_ref="Gate_7_Q0Q0IHN1Yg..divider_2" location="2.41168125519337"/>
  </gating:Quadrant>
  <data-type:custom_info>
   <cytobank>
    <name>CD4 sub</name>
    <id>7</id>
    <gate_id>7</gate_id>
    <type>QuadrantGate</type>
    <version>1</version>
    <fcs_file_id></fcs_file_id>
    <fcs_file_filename></fcs_file_filename>
    <definition>{"scale":{"x":{"flag":4,"argument":"150","min":-200,"max":262144,"bins":256,"size":256},"y":{"flag":4,"argument":"150","min":-200,"max":262144,"bins":256,"size":256}}}</definition>
   </cytobank>
  </data-type:custom_info>
 </gating:QuadrantGate>

Here I also attached the complete example xml generated by cytoML
test.xml.zip
And again the experiment link https://community.cytobank.org/cytobank/experiments/65311

[cciccole]

@mikejiang I agree that the problem seems to be within the JSON definition. The angle field is in radians and is always the same values since we don't currently support skewed quadrants. In any case, it can be ignored. It looks like the minimal amount of information I had to add to your test.xml file within the JSON definition of the quad gate in order to have a successful import was this:

"labels":[[7.015343642234706,7.620918572637606],[0.33115968153503017,7.620918572637606],[0.33115968153503017,-0.3809554276552931],[7.015343642234706,-0.3809554276552931]],"quadrant":{"x":2.23594047371882,"y":3.41168125519337,"UR":1,"UL":2,"LL":3,"LR":4}

Thus making the full definition look like this:
<definition>{"scale":{"x":{"flag":4,"argument":"150","min":-200.0,"max":262144.0,"bins":256,"size":256},"y":{"flag":4,"argument":"150","min":-200.0,"max":262144.0,"bins":256,"size":256}},"labels":[[7.015343642234706,7.620918572637606],[0.33115968153503017,7.620918572637606],[0.33115968153503017,-0.3809554276552931],[7.015343642234706,-0.3809554276552931]],"quadrant":{"x":2.23594047371882,"y":3.41168125519337,"UR":1,"UL":2,"LL":3,"LR":4}}</definition>

Note that the X and Y coordinates in the end of the JSON definition appear to be what control the placement of the gate's center point and the ones found within the divider objects are not used. Nonetheless, they are still required and thus the best practice would be to simply duplicate the information across both places. I haven't verified this 100%, but it appears to me as though the correspondence between the X and Y attributes in the JSON definition and the channel information found in the divider objects is assumed by order of appearance. The first divider object corresponds to X and the second to Y.

It indeed has different requirements than other gates. Yet another level of strange nuance in this system... I'm impressed at how well you've dealt with all of it!

You might run into another problem where gate labels say NA after import (and also the population tree doesn't load). This is actually Cytobank's fault and it's in the process of getting fixed. It has to do with importing a gatingML multiple times that contains identical spill information by name (or something along those lines...still investigating). Let me know (by email) if this afflicts you and I can rescue it on a per-case basis after gates have been imported.

[mikejiang]

Thanks, @cciccole . I also found out that cytobank expects the id or 'gate_id' of the Quadrant is smaller than the one of its parent node QuadrantGate, otherwise, funny thing will happen to the population tree or cytobank will simply error out during importing.

Anyway, quadGate seems to be working now.

[cciccole]

@mikejiang Mm yeah I believe that is indeed another requirement. I didn't experience any error uploading the gatingML you provided, so perhaps in my case there was just strange behavior with the tree and/or stats. I didn't validate those components in my situation (because I didn't know the expected truth), just that things appeared to import okay without error. Thanks for catching this downstream issue. Will try the new code.

[mikejiang]

@cciccole , I am running into the issue when I export the second the quadGate under the same parent but with different x,y dimensions. Cytobank parses the xml without issues, but gives this at the gating page

and gates are visible but not the data ,
I can't figure out what went wrong without detailed the error information from cytobank. So could you please look into that?
here is the experiment link and reproducible code
(note that if you remove one quadgate by uncommenting the lines of code, it will succeed)

library(CytoML)
dataDir <- system.file("extdata",package="flowWorkspaceData")
gs <- load_gs(list.files(dataDir, pattern = "gs_manual",full = TRUE))
Rm("CD8", gs)
#nodes <- getChildren(gs, "CD4")[1:4]
#for(i in nodes)
  #Rm(i, gs)
#output to cytobank
outFile <- "~/cytobank/test.xml"
GatingSet2cytobank(gs, outFile)

[cciccole]

@katdrake to take over since I'm moving on from cytobank

[tjburns08]

@mikejiang I am going to continue this thread to report a recent development with quad gate import. Rather than producing giant rectangles, we get the following upon import into Cytobank:

Here is what the error looks like close up:
"IOException: Cannot create prepared statement for invalid gateId: -3 for gate: <QuadGate id: 0, gateId: -3, Version: -1, Experiment id: 3644, name: "root sub UR", fcsFileId: null, tailored: false, tailoredPerPopulationGateSetId: null, tailoredPerPopulation: false, Compensation Id: Compensation Id: 363, xNormalizedShortNameId: 8, yNormalizedShortNameId: 10>"

I have shown that this error exists in quad gates for both scatter (eg FSC x SSC) and the traditional antibody markers (eg. CD33 x CD3). I have also shown that GatingML from older versions of Cytobank do in fact import into Cytobank, shown below:

I tried looking through GatingML of quad gates from old versus new versions, but it's a bit more complicated than last time.

Luckily, we at least have that more descriptive error message. I am waiting to hear back from Cytobank to see if they have more insight about this error server side. I will email you the wsp, xml, and script to reproduce the error.

[mikejiang]

@tjburns08 , I've pushed the fix, which I believe should fix the error you reported. However, I keep getting the gate plotting error

when I tried to two sets of quad gates under the same parent using different dimensions.
e.g.

This error goes away when I remove either of quadgates before importing to Cytobank
Could you please help with troubleshooting? Because it is difficult for me to do so without more detailed descriptive error message

[mikejiang]

@tjburns08 , Never mind, I've sorted it out on my own. It has to do with the gate_id for 4 quadrants of QuadGate, even though they do have custom_info to store gate_id attribute explicitly in gatingML, Cytobank do expect the these gate_id pre-allocated to them and refer it later on in GateSets definition.
Anyway, Here is the result (not sure why the first quadgate is hidden though)

[tjburns08]

@mikejiang I confirm that the new quad gate architecture works on my end.

I tested the new update on curly quad gate and spider gates and both are still viewed as rectangles, extending out in the southwest direction, both in FlowWorkspace and in Cytobank.

I also note that for some reason, I can't get the percentages to be viewed on my end within Cytobank, though they are displayed on FlowWorkspace. I've emailed you the wsp, fcs, and xml files in the above examples. Thanks for your time.

[mikejiang]

I am not aware that Cytobank also has curlyQuad and spider gate. Can you tell me how to create one in Cytobank? (I don't see this option in gating page)
Regarding to stats, I created a separate issue #28 for you to comment on

[tjburns08]

We can't make them directly within Cytobank, but I found that Cytobank does recognize them and display them. I can push to make this functionality possible if enough users want them.