Standardizing functionality API

[nsheff]

Today we discussed retrieve_multiple, and we're kind of getting a bit lost in the complexity of the API. Here's my attempt to figure out what we need:

• list all record identifiers, paged (list_record_identifiers)
• list all records, paged (list_records)
• get one record (retrieve_one ?)
• get multiple records (retrieve_multiple ?) -- paged? probably not.
• select column filtering?

API interface ideas

In #99 I proposed using the __getitem__ interface to retrieve records. Given the above distinction between retrieve_one and retrieve_multiple we're proposing, one possibility is to allow getitem to be the one that can handle either, like this:

def retrieve_one(record_identifier: str, ...) -> dict
def retrieve_multiple(record_identifier_list: list[str], ...) -> list[dict]
def __getitem__(record_identifiers: Union[str, list[str]], ...) -> Union[dict, list[dict])

So the actual functions have strict schema for parameters and results, but getitem allows for convenience, whereby you can do either psm["xyz123md5"] or psm[["xyz123md5", "abc456md5"]].

Cursor-based pagination ideas

If we implement cursor-based paging following some google standards, we could have an interface like this:

def list_record_identifiers(page_token: str, page_size: int)
def list_records(page_token: str, page_size: int)

return value would be:

{
  total_size: 100,
  page_size: 50,
  page_token: "xyz123md5",  # maybe?
  next_page_token: "gyx456md5",
  prev_page_token: "",  # maybe?
  records: [ ... ]
}

[donaldcampbelljr]

Currently, for retrieving or selecting records, we have 10 functions:

get_one_record
get_records
list_recent_results
list_results
retrieve
retrieve_distinct
retrieve_multiple
select
select_txt
select_distinct

[donaldcampbelljr]

Some idea's from today's discussion:

---

select_records

select_records -> will replace select, select_txt

• can take a list of tuples such that they can be used to filter e.g.
  [(k,v), (k,v), (k,v)] --> distinct columns
  can check type and construct appropriate where statements

if type(k) == str:
    where(k=v)

or

if type(k) == tuple
     where( k = ("genome", "alias", "aa") ==v  )

Can potentially implement OR logic as well:

if type(v) == tuple
   .or(k=v1)
   .or(k=v2)
######
#or:
if type(v) == tuple:
 .in_([v1, v2,...])

---

retrieve_one and retrieve_many

Two convenient retrieval functions which will actually use select_records:
retrieve_one -> retrieving one record
retrieve_many -> retrieving multiple records

---

select_distinct

select_distinct_values -> will be select_distinct, are we able to simply add .distinct to our query from select_records to get the distinct records

---

All of these are two be implemented on the abstract class for BOTH backends.

[donaldcampbelljr]

Ok, first pass at this, implemented:
select_records
retrieve_one
retrieve_many

All three currently return:

        records_dict = {
            "total_size": total_count,
            "page_size": limit,
            "next_page_token": next_cursor,
            "records": results,
        }

select_records is very similar to the original select function but uses cursor pagination.

Next steps:

• Need to change the creation of the dynamic filter as it is not what we want based on yesterday's discussion, i.e. attempt to implement select_txt functionality as well as select_distinct
• retrieve_many loops over a list and performs a select_records call for each r_id, but we should be able to do this in one session call

Also, I left all the original functions in place for now. Will remove them once we are happy with the newest changes.

[nsheff]

this sounds like great progress!

[donaldcampbelljr]

PR merged, this work is now complete.