Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

xAPI - add video tracking #40

Open
garemoko opened this issue Jan 2, 2017 · 46 comments
Open

xAPI - add video tracking #40

garemoko opened this issue Jan 2, 2017 · 46 comments

Comments

@garemoko
Copy link

garemoko commented Jan 2, 2017

It would be great if playing, pausing and watching the video were tracked with xAPI.

See the video recipe below:

Recipe: Video Base (v1)

Go to Details
This recipe defines the structure and terms used in Statements intended to record the experience of watching a video. For comments or suggestions on this recipe please email support+recipes@tincanapi.com.

Basics
All statements include the Recipe ID in the 'category' context activity list.
All statements include an Agent for the watcher as the 'actor' property.
All statements include the video Activity as the 'object'.
Video Activity Definition includes the following:
Has the 'type': http://activitystrea.ms/schema/1.0/video
If available include the length of the video in 'extensions' using: http://id.tincanapi.com/extension/duration
Statements
Play
Sent when the video starts to play either during initial load or after a pause.

Verb: http://activitystrea.ms/schema/1.0/play

Includes the time of the point in the video where playing was started as a context extension, value is a float or an ISO8601 duration.

Extension: http://id.tincanapi.com/extension/starting-point

Pause
Sent when the video is paused, either by active pausing of the playing video or as part of a skip sequence, or reaching the end.

Verb: http://id.tincanapi.com/verb/paused

Includes the time of the point in the video where playing was stopped as a context extension, value is a float or an ISO8601 duration.

Extension: http://id.tincanapi.com/extension/ending-point

Watch
Sent when the video is paused when the previous state of the video was playing in order to capture the segment that was played.

Verb: http://activitystrea.ms/schema/1.0/watch

Includes the time of the point in the video where playing started as well as where it stops using context extensions, both values are floats or ISO8601 durations.

Extensions: http://id.tincanapi.com/extension/starting-point, http://id.tincanapi.com/extension/ending-point

Skip
Sent when the video is paused when the previous state of the video was paused in order to capture the segment that is being skipped.

Verb: http://id.tincanapi.com/verb/skipped

Includes the time of the initial point in the video already reached as well as where playing would resume using context extensions, both values are floats or ISO8601 durations.

Extensions: http://id.tincanapi.com/extension/starting-point, http://id.tincanapi.com/extension/ending-point

Complete
Sent when the video advances to the end.

Verb: http://activitystrea.ms/schema/1.0/complete

Includes the ending time of the video as a context extension, value is a float or an ISO8601 duration.

Extension: http://id.tincanapi.com/extension/ending-point
@fnoks
Copy link
Contributor

fnoks commented Jan 3, 2017

Great. This should be implemented by H5P.Video, which is the generic video-component used by InteractiveVideo.

One problem with this recipe is the "complete"-verb. Often, a viewer may abort/stop/leave the video when there are a few seconds left because there are e.g. closing credits or something similar on the end. So even if the viewer has seen every relevant part of the video, a "complete" event is not sent in this case. Could it be there should be an option for the author to configure when he/she considers the video to be completed?

For future reference: https://registry.tincanapi.com/#profile/19/recipes

@garemoko
Copy link
Author

garemoko commented Jan 3, 2017 •
edited
Loading

@fnoks that recipe defines http://activitystrea.ms/schema/1.0/complete as getting to the end of the video (even if you skipped to the end).

I encourage people to use http://adlnet.gov/expapi/verbs/completed to mean 'completed the video as a learning experience' which could be author configurable and more subjective. H5P could send both.

@fnoks
Copy link
Contributor

fnoks commented Jan 3, 2017

Thank you for clarifying. I have added an issue for sending these events from H5P.Video:
https://h5ptechnology.atlassian.net/browse/HFP-434

@nadavkav
Copy link
Contributor

nadavkav commented Feb 7, 2017

Was kindly pointed here by a colleague (Beni) after opening a feature request at:
h5p/h5p-video#10
And would like to add what seems to be a more detailed xAPI video profile vocabulary:
http://xapi.vocab.pub/datasets/video/

@KirkJohnson
Copy link

Has there been any progress on this? I work for a development group attached to the University of Hawaii at Manoā and we are going to be using H5P interactive videos on a project, but do need more interactions tracked. If no one has added these yet is it possible for us to get a pull request, develop, and submit these to be included?

@fnoks
Copy link
Contributor

fnoks commented Nov 6, 2017

@KirkJohnson: Unfortunately, adding more xAPI statements related to Video is not part of the current plan for the H5P core team. Not that we don't see the need, but we just need to prioritize other tasks. If you are interested in contributing, please create a pull request. I believe this belongs to the H5P.Video library. By putting the logic here, it will be generic to all usage of video inside H5P, not only for Interactive Video.

@KirkJohnson
Copy link

KirkJohnson commented Nov 6, 2017 via email

@nadavkav
Copy link
Contributor

nadavkav commented Nov 6, 2017 •
edited
Loading

I remember @falcon-git wrote some code for supporting xapi video profile during the H5P Conference. is that getting into core?

Also, pointing out to an interesting article and discussion (see the comments) I had with @garemoko a few weeks back.

@garemoko
Copy link
Author

garemoko commented Nov 7, 2017

Worth noting that since that discussion the adl video profile have now published a demo profile: https://www.nextsoftwaresolutions.com/xapi-video-profile-demo/

I've not had time to look at it myself yet.

@nadavkav
Copy link
Contributor

nadavkav commented Nov 7, 2017

Thank you @garemoko for the video profile demo link. very interesting!
Attaching a screen capture of a very promising LRS view of the xAPI video statements:
image

@KirkJohnson
Copy link

KirkJohnson commented Nov 14, 2017 via email

@KirkJohnson
Copy link

KirkJohnson commented Nov 14, 2017 via email

@fnoks
Copy link
Contributor

fnoks commented Nov 22, 2017

Since there are sites out there relying on the completed event, we don't wanna remove it. But if you want, you can add the "ended" event.

We are planning to add functionality that lets the author have more control over what e.g. "completed" means, but we do not have an ETA for that yet.

@KirkJohnson
Copy link

KirkJohnson commented Nov 22, 2017 via email

@KirkJohnson KirkJohnson mentioned this issue Nov 23, 2017
Open
@jmkevan
Copy link

jmkevan commented Jan 17, 2018

Aloha Pål.

Checking to see if we can get an update on this pull request. We still have some time, but we are hoping to have this code incorporated so we can rely on it for an upcoming course we are delivering. Please let us know if there are any issues or ways we can help get this in.

Mahalo,

  • Jon

@fnoks
Copy link
Contributor

fnoks commented Jan 30, 2018

@jmkevan : Sorry for the late reply. We have added a task for reviewing and merging this issue in the sprint we have just started. Stay tuned!

@jmkevan
Copy link

jmkevan commented Jan 30, 2018

@fnoks - No worries and thanks for the response. We noticed on the JIRA ticket that the following comment was posted:

The estimate is high enough to do this all over again.

That seems to imply that the code we submitted was not implemented in the way you would have wanted it. Can we get a little more information about the issue? We are completely fine with you all solving it your own way or if you want help please let us know what could have been better. We want to participate in this community if we can!

I received another email from the video CoP today hoping to move forward with H5P. I will cc' you on that separately.

  • Jon

@falcon-git
Copy link
Member

@jmkevan that was my comment, left in a hurry, should have been phrased differently. I wasn't in the estimation meeting so I didn't know why the estimate was so high. The comment was meant to explain why I postponed the review of the pull request and trigger a response from the devs who estimated it.

I've talked to those who estimated today and also had another look at the PR. The estimate was for us carrying the PR(fix any issues with it), and based on the number of lines changed, the complexity and the amount of testing required. There are device specific and browser specific issues here that I wasn't aware of. We've had problems with Android's HTML5 Video implementation in the past with some of the APIs not working according to the spec. The code looks like a good job, easy to read and well organized.

We're very glad that you're willing to work on this and I've done a very quick review and have three questions before we do a more thorough review:

  1. Is the core change necessary? I thought since the verbs you're adding are not listed here: https://github.com/adlnet-archive/xAPIVerbs/blob/master/verbs.js it would make more sense for you to create a custom verb(define the entire verb object) and it would go into the else here in core:

H5P.XAPIEvent.prototype.setVerb = function (verb) { if (H5P.jQuery.inArray(verb, H5P.XAPIEvent.allowedXAPIVerbs) !== -1) { this.data.statement.verb = { 'id': 'http://adlnet.gov/expapi/verbs/' + verb, 'display': { 'en-US': verb } }; } else if (verb.id !== undefined) { this.data.statement.verb = verb; } };

The core change probably makes the verb incorrect since it gets the http://adlnet.gov/expapi/verbs/ prefix for the verb id?

  1. This code seems to be repeated many times. Maybe it could be a base that is extended for the various events?

"context": { "contextActivities": { "category": [{ "id": "https://w3id.org/xapi/video" }] }, "extensions": { "https://w3id.org/xapi/video/extensions/session-id": Video.sessionID }

Side question: Do you know what the session-id is used for? I see the documentation here, but I don't see why we need a specific session id for video? http://xapi.vocab.pub/describe/?url=https%3A%2F%2Fw3id.org%2Fxapi%2Fvideo%2Fextensions%2Fsession-id

  1. Would it be better to put the xAPI statement generation in a separate file?

@jmkevan
Copy link

jmkevan commented Jan 31, 2018

@falcon-git thanks for the thorough response.

I will let @KirkJohnson respond to most of your questions, but I can answer the side question.

The session-id enables us to get a sense of the frequency of content interaction and not just the quantity of it. That information can then be incorporated into some automated data analysis. For example, we will be using this data to determine how student learning strategies with course content change over time. This can then inform instructional design changes and tools for instructors to find struggling students.

It's also used for visualizations like this heat-map:
https://www.nextsoftwaresolutions.com/xapi-video-profile-demo/

  • Jon

@falcon-git
Copy link
Member

@jmkevan Thanks for answering. I thought context.registration were supposed to be used for things like that, but I see that it isn't very clearly defined in the spec. It would make sense if there were a more "global" property for a session id since video isn't the only activity type that could use a session id.

Looking forward to hear more from @KirkJohnson

@figureone
Copy link

Aloha @falcon-git,

Replying here with @KirkJohnson:

  1. Is the core change necessary?

You're right, in order to use the correct verb URIs for some of the video verbs, we should extend the setVerb() function you mentioned. We'll get started on that.

For context: Our pull requests implement 6 verbs related to video events:

You can see these verbs defined by the Video CoP here: https://github.com/liveaspankaj/xapi-video-cop/blob/master/statement_data_model.md

The current pull request to h5p-php-library simply adds played, paused, seeked, and finished to the list of allowed verbs:
https://github.com/h5p/h5p-php-library/pull/43/files

We will redo that pull request and implement, but there's one wrinkle first: completed vs. finished.

@jmkevan and @falcon-git, to move forward, we need agreement between H5P and the Video CoP on what verb (and URI) to use here. The Video CoP spec uses completed but there is duplication of that verb's meaning elsewhere in the H5P ecosystem.
https://github.com/liveaspankaj/xapi-video-cop/blob/master/statement_data_model.md#user-content-verbs_completed
I believe we agreed on finished, but the spec should be updated, and a verb URI defined (probably https://w3id.org/xapi/video/verbs/finished).

  1. This code seems to be repeated many times. Maybe it could be a base that is extended for the various events?

We thought it better to keep the bulk of the XAPI statement's JSON blob together in one block so a quick review can show what the different video verbs use.

Since context.contextActivities.category.id = "https://w3id.org/xapi/video" seems to exist in every video verb, we could refactor that to h5p-php-library:h5p-x-api-event.js, but I actually don't see contextActivities defined in the spec itself: https://github.com/liveaspankaj/xapi-video-cop/blob/master/statement_data_model.md
@jmkevan, if you could check with the Video CoP to see if that needs to be added to the spec. We added it because it was in the proof of concept by @jhaag75, for example: https://github.com/jhaag75/xapi-videojs/blob/master/xapi-videojs.js#L225-L231

Refactoring session-id would be messier, since we don't have the value for the session ID until we get into the actual video object, so we'd need some guidance from H5P folks if you want it set it in h5p-php-library:h5p-x-api-event.js:setVerb() (current code there only creates constants, not any variables).

  1. Would it be better to put the xAPI statement generation in a separate file?

We'd have no problem with this, but defer to your choice as maintainers (don't want to pollute the namespace unless you sign off on it). Adding them to the Video class made sense to us, but that class is getting large and I could see moving xAPI functionality to a separate class where possible. Just let us know what your decision is.

@figureone
Copy link

figureone commented Feb 7, 2018 •
edited
Loading

Here's that change mentioned in (1) above for reference: uhm-coe/h5p-php-library@d71ed1a

@falcon-git
Copy link
Member

Aloha @figureone and @KirkJohnson,

  1. Is the core change necessary?

I don't think you need to change core at all. The list of ADL verbs in core is just a convenience feature for verbs that are rapidly used. Libraries don't have to provide the entire json for them. It was not my attention that more specific verbs should be added to that list. It isn't a whitelist.

I think using the completed verb is perfectly fine here instead of finished. It will be up to the parent to handle it. i.e. Interactive Video for instance should make sure that the video is modeled as sub-content of the interactive video so that it is clear that a completed from the video sub-content does not mean that the interactive video is complete.

This code seems to be repeated many times. Maybe it could be a base that is extended for the various events?

contextActivities is part of the xAPI spec, and even if it isn't in the video profile used here it can still be uses afaik.

We'll avoid changing core if we don't have to. I think the session-id can stay for now, and we can consider if it is needed if and when core adds a session-id to all statements.

Would it be better to put the xAPI statement generation in a separate file?

Ok, I think we should at least put the statement generation into a separate file.

@KirkJohnson
Copy link

@falcon-git You are the maintainer of the code so whatever choice you make and how you want this to work we leave up to you and trust your judgment. We made the best choices we could think of given our limited knowledge of the code base, but we are not opposed to things changing.

@jhaag75
Copy link

jhaag75 commented Feb 7, 2018

Including @jmkevan @liveaspankaj on this. Thanks gents for the mention. My replies are below. I would totally love to see us work together so the xAPI video profile is supported consistently with H5P.

  1. RE: completed vs. finished.
    I think the Video CoP would gladly add a new verb to help consistency and interoperability. We also recently discussed "ended" and "concluded" as an option. The verb "ended' seems more appropriate for a passive video player activity than a learner though. I guess we just need to come to some consensus on completed. During our last CoP meeting we discussed leaving it to be generic to accommodate the H5P use and allow completed to be send when the video ended. We determined we can key off the progress result extension (https://w3id.org/xapi/video/extensions/progress) to determine whether all of the video had actually been played, rather than coupling it with the "completed" verb.

  2. context.contextActivities.category.id = "https://w3id.org/xapi/video". This is how statements are identified as following a profile. It should be updated in @liveaspankaj documentation. So if you do plan to update H5P to adopt the video profile, rules then you should add it. You could define an H5P specific profile on top of the existing video profile if there are things your player would like to track that we didn't define as part of the video CoP.

@liveaspankaj
Copy link

Hi @jhaag75, I believe we decided to use object.definition.type and droped the context.contextActivities.category.id at some point in the Video Profile.

Also if I am reading @figureone correctly. He is ready to use:

  1. finished verb when user reaches the end of the video, even if he jumps to the end.
  2. completed verb when user completes all parts of the video.

Pankaj

@liveaspankaj
Copy link

@jhaag75 I am a little confused. Probably we needed to remove object.definition.type and keep context.contextActivities.category.id? You are the authority on this part.

@KirkJohnson
Copy link

Hi @liveaspankaj currently in H5P they send a completed verb when the end of the video is reached. This can be triggered either by skipping to the end or playing the video completely. Previously in work our team had done on other projects we only sent completed if the video had been watched fully and based the sending on progress. I think the H5P folks, understandably, are not willing to have completed be implemented differently since there are already applications relying on how they are defining completed so that is what prompted the discussion on adding a new verb which would encompass not only the video finishing, but the user having watched all of it as well. Anyone please feel free to correct me if I'm misunderstanding the situation.

@jmkevan
Copy link

jmkevan commented Feb 8, 2018

You are on point @KirkJohnson .

We are still trying to gather a consensus on a solution to the completed issue. There are essentially three options being discussed:

  1. H5P aligns with the video profile. Your group has already said they are unwilling to do that due to existing practice relying on completed meaning 'reached the end of the video'. We can appreciate this stance.
  2. We send two 'completed' statements assuming this is possible. If so, we have added the contextActivities.category.id information to the video profile. All video profile compliant statements MUST contain this now which enables profile users to easily filter out H5P 'completed' statements (or any other non-compliant datum). We will update our code as necessary to include the relevant lines.
  3. We change the video profile term for 'completed' to something like 'ended' per @jhaag75 's comment. The community may be leaning away from this now though. They are concerned about having to make changes to the profile every time there is a mismatch.

Would H5P be fine with #2?

As a side note, there is an issue we will be looking into in a future PR. Students don't always watch exactly 100% (could be 99%), so we are discussing the ability to set a threshold value somewhere. We are not sure where this value should be exposed though. If you happen to have any thoughts on that please throw it our way. Like I said, this will be a future change though.

@jmkevan
Copy link

jmkevan commented Feb 8, 2018

That's all assuming there are no perceived issue from @KirkJohnson or @figureone whom I have not reviewed that information with yet!

@falcon-git
Copy link
Member

Regarding completed vs. finished we do whatever the profile says. It doesn't matter much to H5P. Most of our xAPI coverage is for content types that has questions and results. Video doesn't.

For interactive video we're now adding a submit screen that will result in a completed event for the interactive video. If the video sub content of the interactive video also triggers a completed event it won't be a problem since H5P won't interpret it as the Interactive Video being completed and an LRS shouldn't either. If we don't model the video as sub content a completed event from the video will be changed to something else by interactive video if the interactive video has interactions and a submit screen.

If the video isn't modeled as sub content will it be correct to say that the interactive video follows the video profile? Can an activity follow multiple profiles? The video profile isn't made for interactive videos it seems.

@jhaag75
Copy link

jhaag75 commented Feb 8, 2018

@falcon-git Definitely. An activity can use multiple profiles!

@liveaspankaj Yes, we must set the context category id. That's mandatory. The object definition is optional, but a good practice too.

@jmkevan Yes, #2 + #3 sounds good but we can keep the "completed" verb. We could also add "finished" or "ended" to represent the video was completed without guarantee that there was any interaction. I think during our last meeting @liveaspankaj said we could just key off of the progress extension to determine if 100% was completed.

@liveaspankaj
Copy link

I believe @falcon-git is suggesting that we use Video activity as "sub activity" to the Interactive Video activity (H5P Activity).

And use the Video Profile as it is currently, including the completion on consuming all parts of the video.

If that is acceptable to all (@falcon-git, @KirkJohnson, @jhaag75, @jmkevan ) I guess there is no other pending conflicts for the moment.

@falcon-git, You are right, Video Profile is not made for Interactive Videos, but we did discuss it and agreed that any sub activity statements can be added with acivity id of video as parent id to create the linkage. However, looking at your use case, I guess there is scope for more discussion on including interactive components, specially if the completion of interactive elements effect the overall completion. Using video as sub activity could work as a good immediate term solution.

Regards,
Pankaj Agrawal

@jmkevan
Copy link

jmkevan commented Feb 8, 2018

Yes, that's my understanding as well @liveaspankaj ! That's great 👍

In summation...

  1. H5P is comfortable with us implementing and intends to adopt the xAPI Video Profile in its entirety as part of the video component. This includes our verbs, definition of completed and any other relevant statement components (like progress).
  2. We need to move the xAPI Statement Generation in our PR to a new file. @falcon-git can you please clarify which files have the content you want us to move (just so we don't misinterpret) and where the new statement generation file should live?
  3. In a future PR we will deal with video completed statements where progress is close to but not exactly 1.0.
  4. The Video CoP will discuss next steps for an Interactive Video Profile or other solution.

Let me know if I am missing something.

  • Jon

@falcon-git
Copy link
Member

@liveaspankaj I have a couple of concerns though when I think about it. We've had requests for being able to add video interactions on top of the Interactive Video. If we do there will initially not be any difference between those and the main video in the statement, maybe except that the interactions will have information about where in the video they are whereas the main video won't. There is definitively a need for an interactive video profile. Can a profile have rules about events from sub content? Would be cool if a profile could inherit from another and override. The interactive video profile could inherit video, override the completion part and add some rules for interactions within the video if applicable.

@jmkevan Even if there is questions regarding interactive video, all issues regarding H5P video should be clear and good to go! Your summation:

  1. Cool
  2. Live in the H5P Video library and moves code out of video.js into x-api.js. Clear candidates are guid and all the getArgsXAPI..., the logic for keeping track of sections played maybe doesn't make that much sense to move? You decide.
  3. Cool. I think that will be important.
  4. Cool

@jhaag75
Copy link

jhaag75 commented Feb 9, 2018

Adding @fugu13 to this thread in case he has any perspectives about mixing profiles.

@liveaspankaj
Copy link

liveaspankaj commented Feb 9, 2018 via email

@jmkevan
Copy link

jmkevan commented Feb 9, 2018

Thanks @falcon-git .

@liveaspankaj - Hopefully I can bring some practical knowledge to the Video CoP once we are done with this Fall course delivery using the H5P interactive videos. I am also exploring its use in a food innovation program at Maui College.. so more potential use cases to draw from.

I agree that most of the interactive components fall within the purview of existing or larger profiles. The assessment questions and social functionality are good examples. There may be some exceptions such as having supplementary content (or a link to) pop-up during a video. That's such a generic instructional strategy or event that I am not sure any broader profile makes sense for that.

Either way, that's an issue our group would be willing to tackle and think through after our course is delivered if it doesn't get solved before then.

  • Jon

@jmkevan
Copy link

jmkevan commented Feb 9, 2018

@falcon-git btw, one of our developers is out of town.. so we wont get to these updates for a few weeks. We will reach out as soon as we have something to share.

@liveaspankaj
Copy link

liveaspankaj commented Feb 10, 2018 via email

@jhaag75
Copy link

jhaag75 commented Feb 12, 2018 via email

@jhaag75
Copy link

jhaag75 commented Feb 12, 2018 via email

@falcon-git
Copy link
Member

@liveaspankaj you are right, it is probably quite wide, but there are at least a couple of things that are common for most of the interactive video implementations I've seen, ordered by how common things are:

  1. The interactions appear at a certain time in the video and disappear at a certain time
  2. The interactions may or may not pause the video when they appear
  3. Invisible hotspots on top of the video that when click send the user to a url or a nother time in the video or loads another video.
  4. Interactions may also often end with the user going to another place in the video.
  5. Navigations through various interactions may form different paths through the video so that it won't be possible for a learner to see the entire video unless there is a path that covers the entire video

The interactions themselves are modeled as sub content by H5P in the xAPI statements, and I think they should. And they typically already follow established profiles but I think they should contain info about where in the video the user where when interacting with the sub content and also where the sub content starts and stops in the video.

@liveaspankaj
Copy link

liveaspankaj commented Feb 13, 2018 via email

@figureone figureone mentioned this issue Feb 23, 2018
Closed
@figureone
Copy link

Aloha @falcon-git, the pull request is ready for review again:
h5p/h5p-video#17

xAPI features have been refactored to x-api.js and test xAPI statements submitted to an LRS have been reviewed and confirmed to match the Video Profile by @jmkevan.

@falcon-git
Copy link
Member

Great! Wow, this is a huge pull request! Will be interesting to see this in action with a suitable LRS. I'm happy with the big lines now, I guess the code that still is in html5 and youtube scripts need to be there. Another core team member will do a more thorough review soon. I've updated the Jira issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@figureone @nadavkav @jhaag75 @falcon-git @garemoko @liveaspankaj @fnoks @jmkevan @KirkJohnson