All messages in a room always returned by messages.list() method

[further-reading]

When I run the messages.list() method it returns all messages in a room even with a max argument.

Sample code

from ciscosparkapi import CiscoSparkAPI

api = CiscoSparkAPI()

messages = api.messages.list(roomId="ROOMID", max=10)

Expected: 10 messages returned
Actual: every message in the room is returned.

[cmlccie]

@further-reading : Per the API spec the max parameter specifies the maximum number of items per page ("page size").

Since ciscosparkapi transparently handles paging for you… It respectfully uses your requested page size, and still returns the all of the results - requesting additional pages as needed until all results have been returned. 😁

If you only want a certain number of items returned, you can use the Python standard library's itertools.islice() function to get just what you need:

from itertools import islice

from ciscosparkapi import CiscoSparkAPI

api = CiscoSparkAPI()

messages = api.messages.list(roomId='<your-room-id>')

# Only want the first 50 messages?
for message in islice(messages, 50):
    print(message)

# Want them in a list?
short_list = list(islice(messages, 50))

I have considered adding a convenience slicing ability to the package's GeneratorContainers, which would effectively do the very same thing only with the simple syntax of messages[:50].

[further-reading]

Thanks for the explanation! :)

Just to clarify, say I do something like this in a room with 1000 messages:

from itertools import islice
from ciscosparkapi import CiscoSparkAPI

api = CiscoSparkAPI()

messages = api.messages.list(roomId='<your-room-id>', max=10)

#get 11
for message in islice(messages, 11):
    print(message)

Which best describes what the ciscosparkapi package does:

1. Detects there are additional pages on the initial response, immediately goes through all available pages until all 1000 messages have been fetched.
2. Fetches the first page worth of messages only. When the loop reaches the 11th message it will then fetch the 2nd page.

In other words, does ciscosparkapi fetch all messages immediately or will it wait until the additional pages are needed first?

[SomeClown]

Looks like the actual endpoint is working fine, it's just the CiscoSparkAPI wrapper that is ignoring the max parameter. The code below works as expected, only returning 2 messages.

import requests

def send_request():
    try:
        response = requests.get(
            url="https://api.ciscospark.com/v1/messages",
            params={
                "roomId": "<my_room_id>",
                "max": "2",
            },
            headers={
                "Authorization": "Bearer <my_token>",
            },
        )
        print('Response HTTP Status Code: {status_code}'.format(
            status_code=response.status_code))
        print('Response HTTP Response Body: {content}'.format(
            content=response.content))
    except requests.exceptions.RequestException:
        print('HTTP Request failed')

[further-reading]

Based on what @cmlccie said, despite what the api docs say the max parameter is actually setting page size not the maximum returned results. So when requests is used it returns the first page, but CiscoSparkAPI will see that there are more pages and is capable of walking over them to get everything.

I did some testing earlier to confirm. If you don't set a max the most requests will return 99, which is the default page length.

[SomeClown]

Right, but using the requests module and hitting the endpoint directly, if I put in a max=1 I get one item, equal to one message. If I put in 5, I get 5 items, etc. If I put in arbitrary numbers above 100 I get all of those as well (so, 143 or 229). If I don't use max at all I get 49 in the room I'm testing.

Keeping in mind this is not with the Python wrapper API, this is just directly with the code above.

[further-reading]

To clarify where the Cisco docs are confusing:
From list messages

Limit the maximum number of messages in the response.

But pagination clarifies:

When requesting a list of resources the max query parameter may be used to control the number of items returned per page.

So like I said, when you use the requests module you are actually getting just the first page of results as the response. The wrapper though can see that there are additional pages available and is capable of requesting those too. In other words, all the max argument does in the wrapper is change how many requests it'd need to make to get everything. If there are 1000 messages and you set max at 1000 the wrapper makes one request, if you set max at 500 it makes two requests.

[bradh11]

The response from Cisco will have a link to the next page in the header when your result set is paginated. You will need to recursively make api calls to the next url found in the header to get the additional results.

…

On Dec 31, 2017 at 7:03 PM, <Roy Healy ***@***.***)> wrote: To clarify where the Cisco docs are confusing: From list messages (https://developer.ciscospark.com/endpoint-messages-get.html) > > > Limit the maximum number of messages in the response. > > But pagination (https://developer.ciscospark.com/pagination.html) clarifies: > > > When requesting a list of resources the max query parameter may be used to control the number of items returned per page. > > So like I said, when you use the requests module you are actually getting just the first page of results as the response. The wrapper though will also see that there are additional pages available and is capable of requesting those too. In other words, all the max argument does in the wrapper is change how many requests it'd need to make to get everything. If there are 1000 messages and you set max at 1000 the wrapper makes one request, if you set max at 500 it makes two requests. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub (#50 (comment)), or mute the thread (https://github.com/notifications/unsubscribe-auth/ARoKxucJHoOk9ciKhCnUGq4l8af-mtZGks5tGCC3gaJpZM4RCovt).

[SomeClown]

So if I'm reading those docs correctly, when I set "max=1" I am getting one result per page, and since I'm not then asking for any additional pages, I'm getting just my one item. I see the link returned for next, so I could ask for more pages, but I'm not (using my example code above). So my main question I guess is how come the wrapper is automagically returning all additional pages? It may be respecting my request for one result per page, but it is then giving me all pages. I don't see a way in the wrapper to limit page results in the message.list method.

So (again, if I'm understanding this correctly) the wrapper is helpfully giving me all pages available, grabbing the next=rel link and doing it using my max=1 parameter. So if I want an efficient way to get only one item, I should use the direct endpoint and skip the wrapper in this one case, otherwise the wrapper is going to give me everything and then I have to slice and dice to get just my one, which would be very inefficient with large result sets.

Is that an accurate summary, or am I missing something?

[SomeClown]

A deeper read of the documentation for the wrapper seems to confirm the above... from: http://ciscosparkapi.readthedocs.io/en/latest/user/api.html#messages

This method supports Cisco Spark’s implementation of RFC5988 Web Linking to provide pagination support. It returns a generator container that incrementally yields all messages returned by the query. The generator will automatically request additional ‘pages’ of responses from Spark as needed until all responses have been returned.

This only became an issue for me because of one particular room having 18,000 messages. The speed difference in requesting 100 messages only, vs. getting all 18,000 and then parsing to only give the user 100, is a giant speed difference on the order of seconds vs. minutes.

[further-reading]

That's accurate, yeah.

The dev suggested some code earlier to grab a specific number of messages from the output of the api call, however I'm not sure about its efficiency. I asked him to clarify at which point the wrapper will request the next pages.

[cmlccie]

Great discussion. 🙂 Efficiency is the primary reason for the creation and use of the GeneratorContainers (that incrementally yield results only as needed/requested by your code).

A Python generator is a great tool for representing the "lists" of items that can be returned by Cisco Spark as they help with the following challenges and more:

• Let's say you are looking for a particular room (find by title). When requesting the list of rooms, you don't know beforehand how many rooms might be needed to find the one that you are looking for, and requesting "all" of the rooms is wasteful (mostly a waste of time). Compare the response time for a list of 10 items to a list of 1,000 items; if what you are looking for can be found in the first 10 rooms (or 20) requesting small incremental pages will speed up the responsiveness of your script.

• What about when you want to look at the messages in a room? How far back do you want to go? Messages from today, since yesterday, this week, this month? How many messages might be represented in those time increments? Trying to request a static number of messages would be a guessing game and error prone (and you would have to deal with paging).

• What if you really do want all of the items (all of the rooms, all of the messages, and etc.)? How many items would you have to request? If it is larger than Spark's default page size maximum, you would have to handle paging. How many pages would it be?

Generator functions provide efficient and flexible functionality to suit a wide variety of use cases:

• If you want to iterate through a "list" of items from the cloud and stop when you have found what you are looking for, you can do that (only the minimally needed pages of items are requested).

for room in api.rooms.list():
    if room.title == "The room I am looking for...":
        found_it = room
        break
# The ciscosparkapi.room.list generator will request a page of items, yielding them one at a time
# to your code (your for loop) requesting additional pages as needed until you either break out of
# the loop (you found what you were looking for) or all items have been returned (at which point
# the for loop will exit.

• If you want some specific number of items... You can do that efficiently.

from itertools import islice
from ciscosparkapi import CiscoSparkAPI

NUMBER_OF_MESSAGES = 2000

api = CiscoSparkAPI()
messages = api.messages.list(roomId='<your-room-id>', max=NUMBER_OF_MESSAGES)

short_list = list(islice(messages, NUMBER_OF_MESSAGES))

# By setting the maximum number of items per page to the total number that you want, if Spark
# will allow sending that many items in a single response - cool you'll get them all in one response. 
# If not, the ciscosparkapi package will transparently handle the requesting of additional pages for
# you, and the Python standard library `islice()` iterator-slicer will ensure that only the requested
# number of items will be yielded and passed to the `list()` function - result you will have a list
# containing up to `NUMBER_OF_MESSAGES` messages.

• If you really want all of the items in a real Python list? Also, easy:

room_list = list(api.rooms.list())

So, no matter how many items you want returned: a few, some in-determinant number of items, or if you want all of the items - a generator function is the perfect single interface to accommodate all of these needs as efficiently as the APIs and your use case will allow. If you need all the items, we'll really have to get them all. If you need some, we'll stop when you tell us. 😎

So, that's what generators do! ...but what's up with the "containers?" Well, the short answer is that an iterator returned by a generator function is a single-use sort of thing. Once you have iterated it, you can't iterate it again. You would have to call the generator function to get another iterator. This is a pain and error prone, so I "wrapped" the generator functions in a container that makes them safe for reuse. Every time you iterate one of our generator containers, the container takes care of creating a "fresh" iterator for you.

[cmlccie]

I added the enhancement keyword to this issue, as I will likely go ahead and add a native slicing feature to to the generator containers so that use cases like yours (where it appears that you want a specific number of items) can be syntactically easier.

What do you think of the following user experience?

rooms = api.rooms.list()

ten_rooms = rooms[:10]
fifty_rooms = rooms[:50]
up_to_one_thousand_rooms = rooms[:1000]
all_rooms = rooms[:]

# These would all be generators yielding items up to the maximum number of items requested
# (obviously if there weren't 1,000 rooms - we would just return all that exist).

Note that for each of the above requests, a new series of requests and paged (if necessary) responses would be initiated. With the ciscosparkapi package, you are always requesting and working with "live" data from the cloud. 😎