Only data for one column is returned when reading row with same qualifier in multiple families

[erifol]

Environment details

• OS: macOS Big Sur
• .NET version: 7
• Package name and version: Google.Cloud.Bigtable.V2 3.3.0

Steps to reproduce

1. Create a table with two column families
2. Create a row with the same column qualifier appearing in both column families
3. Try to read the row - only data for one of the columns is returned

Seems that the issue could be related to RowAsyncEnumerator;

google-cloud-dotnet/apis/Google.Cloud.Bigtable.V2/Google.Cloud.Bigtable.V2/RowAsyncEnumerator.cs

Line 335 in 8e661bb

if (chunk.Qualifier != null && chunk.Qualifier != owner._currentCell.Column?.Qualifier)

. It seems to me that the cell chunk data is not handled if two cell chunks with different families but equal qualifiers appears consecutively.

[jskeet]

Assigning to Rishabh to investigate. (I'd have a look myself, but I'm off sick at the moment.)

[Rishabh-V]

I'll have a look at it first thing tomorrow morning IST and share my investigation. Thanks.

[erifol]

I tried to write a simple program to reproduce the problem. Hope it is helpful.

        var tableId = "TestTable";
        var table = new Table
        {
            Granularity = Table.Types.TimestampGranularity.Millis,
            ColumnFamilies =
            {
                { "MyFamily1", new ColumnFamily { GcRule = new GcRule { MaxNumVersions = 10 } } },
                { "MyFamily2", new ColumnFamily { GcRule = new GcRule { MaxNumVersions = 10 } } }
            }
        };

        try
        {
            await _tableAdminClient.CreateTableAsync(
                parent: new InstanceName(_settings.ProjectId, _settings.InstanceId),
                tableId,
                table);
        }
        catch (Exception)
        {
            // ignored, table already exists
        }

        var tableName = TableName.FromProjectInstanceTable(_settings.ProjectId, _settings.InstanceId, tableId);
        var rowKey = ByteString.CopyFromUtf8("row-key-0");

        var mutateRowRequest = new MutateRowRequest()
        {
            TableNameAsTableName = tableName,
            RowKey = ByteString.CopyFromUtf8("row-key-0"),
            Mutations =
            {
                Mutations.SetCell(
                    "MyFamily1", 
                    ByteString.CopyFromUtf8("my-qualifier"),
                    ByteString.CopyFromUtf8("in-family-1")),
                Mutations.SetCell(
                    "MyFamily2", 
                    ByteString.CopyFromUtf8("my-qualifier"),
                    ByteString.CopyFromUtf8("in-family-2"))
            }
        };

        await _bigtableClient.MutateRowAsync(mutateRowRequest);

        var row = await _bigtableClient.ReadRowAsync(tableName, rowKey);

        foreach (var family in row.Families)
        {
            foreach (var column in family.Columns)
            {
                Console.WriteLine(column.Cells.First().Value.ToStringUtf8());
            }
        }

The program only prints
in-family-1,
but I would expect it to also print 'in-family-2'. If I change the name of the qualifier in the second column family to my-qualifier-2, the program prints
in-family-1
in-family-2

[Rishabh-V]

Right. I wrote a similar sample app and was able to reproduce this issue.

In addition to different qualifiers working, I observed that if there is another cell with a different qualifier in the first family, then also all the column values are printed.

I will further investigate tomorrow and update when I know more about it. Thanks.

[igorbernstein2]

It looks like the problem is that when the family changes on lines:

google-cloud-dotnet/apis/Google.Cloud.Bigtable.V2/Google.Cloud.Bigtable.V2/RowAsyncEnumerator.cs

Lines 324 to 333 in 8e661bb

	if (chunk.FamilyName != null)
	{
	if (chunk.FamilyName != owner._currentCell.Family?.Name)
	{
	owner._currentCell.Family = new Family { Name = chunk.FamilyName };
	Debug.Assert(!owner._currentFamilies.ContainsKey(chunk.FamilyName));
	owner._currentFamilies[chunk.FamilyName] = owner._currentCell.Family;
	}
	owner.Assert(chunk.Qualifier != null, "NewCell has a familyName, but no qualifier");
	}

The _currentCell contents are not fully reset, only the family info is. So the specific tweak for this bug would be to reset currentCell.Column when FamilyName changes. One possible implementation would be: in the Family change if statement body set a flag familyChanged and use that as an or condition in the column if statement condition.

[igorbernstein2]

Also, please add a test in the conformance test json file for this issue:
https://github.com/googleapis/conformance-tests/blob/main/bigtable/v2/readrows.json