WARNING: don't know how to handle filter of type *google_bigtable_v2.RowFilter_RowKeyRegexFilter

[simonmorley]

Running some python units tests against the bigtable emulator, I saw the following error in the logs:

WARNING: don't know how to handle filter of type *google_bigtable_v2.RowFilter_RowKeyRegexFilter

I tried a few different regex patterns w/o success (bytes, string etc). My row key looks like this 3917#88DC961302E8#201703 and the regex 'test-patter' ain't working.

regex = 'test-pattern'
col1_filter = RowKeyRegexFilter(regex)
col2_filter = FamilyNameRegexFilter(b'UNQS')
chain = RowFilterChain(filters=[col1_filter, col2_filter])
partial_rows = table.read_rows(filter_=chain)

Works against production table without issue.

I was running Google Cloud SDK 138.0.0 so I updated and am now running this:

Google Cloud SDK 147.0.0
alpha 2016.01.12
beta 2016.01.12
bigtable
bigtable-darwin-x86_64 2017.01.20
bq 2.0.24
bq-nix 2.0.24
cloud-datastore-emulator 1.2.1
core 2017.03.13
core-nix 2016.11.07
gcd-emulator v1beta3-1.0.0
gcloud
gcloud-deps 2017.03.13
gcloud-deps-darwin-x86_64 2017.02.21
gsutil 4.22
gsutil-nix 4.18
kubectl
kubectl-darwin-x86_64 1.5.3
pubsub-emulator 2017.02.07

The error has now gone from the logs BUT the problem still remains. The filter isn't applying.

[garye]

Can you possibly send a small test case that inserts a row and then tries unsuccessfully to filter for it using a regex? The code you already provided is close but "test-pattern" doesn't seem like it should match the row key so I think I'm missing something.

One thing to note is that the emulator can't handle binary regular expressions. See this document for supported filters and limitations.

Also, the error you originally saw was valid until a month or two ago when support for RowKeyRegexFilter was added.

[simonmorley]

Yeah I saw the binary regex note. Give me a couple of days, am travelling today.

OK about the time it was added. It's not throwing an error at least but might be something else. I'll write a test for it and post the code in a couple.

[simonmorley]

Ok, a much reduced function including test. Should filter by the ap_mac in this case. Works OK on production table. Let me know if I've either done something wrong (!) or if you need more information.

(format_mac just strips the - from the mac address)

    def snr_test(self, obj, group):

        a = []

        ap_mac = self.format_mac(obj['ap_mac'])
        regex = str(obj['location_id'])[::-1] + '#' + ap_mac + '#([0-9]{10})'

        table = instance.table("METRICS")
        column_family_id = 'ST'

        col1_filter = RowKeyRegexFilter(regex)

        partial_rows = table.read_rows(filter_=col1_filter)
        partial_rows.consume_all()

        for row_key, row in partial_rows.rows.items():
            key = row_key.decode('utf-8')
            cell = row.cells[column_family_id]
            timestamp = cell[list(cell.keys())[0]][0].timestamp
            val = { "timestamp": timestamp }
            a.append(val)

        return a

And the test:

    def test_clients_snr_test(self):

        s = datetime.today()

        group = '1D'
        location_id = 7193
        table_id = "METRICS"
        macF = '1122AA223344'
        column_family_id = 'ST'
        column_id = ('SNR:' + macF).encode('utf-8')
        table = self.__class__.instance.table(table_id)

        try:
            table.delete()
        except:
            pass

        table.create()
        cf1 = table.column_family(column_family_id)
        cf1.create()

        keyz = [
            '3917#88DC961302E8#' + s.strftime('%s'),
        ]


        for i, value in enumerate(keyz):
            row_key = value
            row = table.row(row_key)
            row.set_cell(
                column_family_id,
                column_id,
                '123',
                s
                )
            row.commit()

        m = Metric()

        obj = {}
        obj['location_id'] = location_id

        ### Now with correct mac address
        obj['ap_mac'] = '88-DC-96-13-02-E8'
        a = m.snr_test(obj, group)

        self.assertEqual(len(a), 1)

        ### Now with WRONG ap mac
        obj['ap_mac'] = 'XX-XX-XX-XX-XX-XX'
        self.assertEqual(len(a), 0)

        table.delete()

Which fails at the last assert. The regexfilter's not being applied using the emulator.

[garye]

The fix has been released in gcloud for some time.