Add JSON returns

[jleatham]

Would be nice if in addition to the generator container, you offered a method of receiving plain JSON. This would allow more flexibility in data manipulation, e.g., loading spark data into pandas, etc

For instance:
sparkDataJSON = api.rooms.list(format="json")

[cmlccie]

@jleatham As a Pandas user and fan, I am with you here. I have recently been thinking of enhancing the SparkData base class to add the following functionality:

• Internally maintain the JSON attribute ordering by using OrderedDict for the internal SparkData._json data structure.
• Exposing public SparkData.to_dict() and SparkData.to_json() methods to provide direct access to the JSON data (though I would return copies to prevent unintentional manipulation).

Additionally, I am beginning work on the ciscosparksdk package that will extend the functionality of the ciscosparkapi package and provide a number of conveniences and tools for Python Spark developers. To enable the initial functionality, I am going to have to enable the ciscosparkapi API methods to be able to return different object types (in my case sub-classes of the ciscosparkapi classes). I could probably simultaneously enable the API methods to return object types like dict and/or OrderedDict to accomplish your purposes... With one note...

Note: These enhancements would only act on individual SparkData items, not the "macro" contents returned by a method that uses the GeneratorContainers to enable efficient paging.

So, you would have to do something like this:

import pandas as pd
from ciscosparkapi import CiscoSparkAPI

api = CiscoSparkAPI()

# Would result in a list of dictionaries
spark_data = [room.to_dict() for room in api.rooms.list()]

# Which could be used to create a Pandas DataFrame
data_frame = pd.DataFrame(spark_data)

-or-

import pandas as pd
from ciscosparkapi import CiscoSparkAPI

api = CiscoSparkAPI()

# Would result in a list of dictionaries
spark_data = list(api.rooms.list(item_return_type=dict))

# Which could be used to create a Pandas DataFrame
data_frame = pd.DataFrame(spark_data)

Would one of these work for you? Do you have a preference?

[jleatham]

Probably option 2. It would be nice if it automatically put the dict in a list when there were multiples.
This is what the meraki API does(though now that I think about it, that causes some irregularities. Maybe it would be better to just always enclose in a list, even if only one {dataset}.) I got spark working with pandas in a similar way, just with extra steps:

sparkTeamsList = []
sparkTeamsItem = {}
for team in sparkTeams:
      sparkTeamsItem = {'creatorId':team.creatorId,'created':team.created,'name':team.name,'id':team.id}
       sparkTeamsList.append(sparkTeamsItem)

df = pd.DataFrame(sparkTeamsList)

Your way is much nicer! Thanks for taking a look.

[cmlccie]

The only challenge with "automatically put the dict in a list when there were multiples" is that it defeats the purpose for the generators. Think about times when you are "listing" things...

Rooms: If you have a lot of rooms and you are listing the rooms to find a specific one (which usually is a recent one), it can take multiple / many API calls to request 'all' of the rooms. And, if the room is let's say in the first 10 or 20 rooms, all of the following API calls weren't needed.

Messages: What if you requested "the list" of messages for a decently trafficked room that has existed for let's say over a year. That room could contain a lot of messages. If you are only interested in the recent messages, there would be a ton of wasted API calls (and memory to store all of the objects).

The generators, which incrementally yield the results (requesting additional pages only as needed) are more efficient. If you find what you need, you can stop the iteration and not make any more requests than are needed. If you want all of the items, you can of course continue the iteration until all items have been returned - in essence you are in the drivers seat. 😎

Plus, if you really want a list (which will contain all the requested items), you can always just pass the generator container to the list() constructor which will build a list from all of the generated items.

rooms_generator = api.rooms.list()
rooms_list = list(rooms_generator)

[cmlccie]

While I will have to add functionality to the package to enable different item data types to be returned (to enable the ciscosparksdk functionality), I believe I favor the first option. I think it gives the best of both worlds. You still have the benefit of the of the native Python objects, and if you want access to the underlying json_data, you can easily access it.

[cmlccie]

@jleatham I added method number one above in v0.8.2. To accomplish your purpose (importing lists into pandas data frames), you should now be able to simply do the following:

import pandas as pd
from ciscosparkapi import CiscoSparkAPI

api = CiscoSparkAPI()

spark_data = [room.json_data for room in api.rooms.list()]
data_frame = pd.DataFrame(spark_data)

Since I implemented the object's base data type as an OrderedDict and exposed a copy of the JSON data via the json_data property, the response attribute order should be maintained all the way through to your pandas data frames (which means they look nicer - by default).

Let me know if you have any issues. -Thank you!

[jleatham]

I agree that that is the more ‘pythonic’ way of doing it. Thanks for your help. I’ll give it a shot soon. edit: works! Thanks. Looking at the output the JSON doesn't look like the JSON I have worked with in the past , i.e., [{"id":"123","title":"hello"},{"id":"1234","title":"hello2"}]. Instead it is more like: [([("id":"123"),("id":"1234")])] . That being said, I am pretty new to python, and it works with pandas out of the box, so no worries there. Anyways, thanks again!

…

On Sun, Nov 5, 2017 at 1:19 AM Chris Lunsford ***@***.***> wrote: @jleatham <https://github.com/jleatham> I added method number one above in v0.8.2. To accomplish your purpose (importing lists into pandas data frames), you should now be able to simply do the following: import pandas as pdfrom ciscosparkapi import CiscoSparkAPI api = CiscoSparkAPI() spark_data = [room.json_data for room in api.rooms.list()] data_frame = pd.DataFrame(spark_data) Since I implemented the object's base data type as an OrderedDict and exposed a copy of the JSON data via the json_data property, the response attribute order should be maintained all the way through to your pandas data frames (which means they look nicer - by default). Let me know if you have any issues. *-Thank you!* — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#48 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACNtZPGQVr01lTVxtqS-FmXjNanzIanVks5szWDrgaJpZM4QQb4f> .

[cmlccie]

You are correct... The json_data property is exposing a copy of the OrderedDict that we are using internally to represent the parsed JSON data. The OrderedDict class maintains the order in which key:value pairs were added to collection (unlike a regular dict, which is not guaranteed to do so). If you want something that looks more like what you are used to, you can also use the to_dict() and to_json() methods that I added at the same time. The to_dict() method will convert the OrderedDict object into a standard (unordered) dict.

You will find that the OrderedDict's do behave essentially just like dictionaries (the class is actually a subclass of dict) only with the added feature that attribute order is maintained; and yes, they can look a little weird. 😎