Auto-populate Instances for a given Target based on what's in Wayback

[anjackson]

The original intention was that W3ACT would take the 'lead' URL for a site and auto-populate the instances based on what's in Wayback. e.g.

http://www.webarchive.org.uk/wayback/archive/xmlquery.jsp?url=http://www.bl.uk/bibliographic/ukmarc.html

Similarly, our internal Wayback instance has an XML query endpoint that allows you to look up the dates we have instances for.

The idea would be to automatically scan for new instances when a Target is visited, but to also have a special URL we can call every night that checks for new instances of all Targets.

[anjackson]

How's that, Peter?

[peterwebster]

@anjackson wondering whether to dynamically check for each target record when opened by user might not slow things down. Perhaps just on creation of a new record ? So:

(i) User creates new target record -> ACT checks for instances of that target in Wayback > creates new instance record for each

AND

(ii) nightly, ACT checks the Wayback XML endpoint for instances of any targets for which there are not existing instance records (ie checking instances with timestamps since the last check) > creates new instance records for each.

@kinmanli how much work do you think this might be ?

[kinmanli]

@anjackson if i was to deal with a Target with a url "https://www.gov.uk/government/publications". Is the url to wayback

"http://www.webarchive.org.uk/wayback/archive/xmlquery.jsp?url=https://www.gov.uk/government/publications" ????

As it gives me "Resource Not In Archive"

or is it the domain?

"http://www.webarchive.org.uk/wayback/archive/xmlquery.jsp?url=https://www.gov.uk/"

Cheers

[anjackson]

Hello, yes, that was the right URL for the public instance, which you can use for testing. Unfortunately, there was a problem with it, but it's fixed now.

However, note that in production, this should be pointing to our internal QA Wayback (which gets to see much more content much earlier than the public Wayback) so the location of this endpoint should be configurable.

[peterwebster]

@kinmanli I created a new target, saved the record; and then hit Get New Instances.

I know this site, and doubt it is in the public Wayback - so this may have returned no new instances - is that why I get?

Execution exception[[RuntimeException: uk.bl.exception.ActException: javax.xml.bind.UnmarshalException - with linked exception: [java.io.FileNotFoundException: http://www.webarchive.org.uk/wayback/archive/xmlquery.jsp?url=http://www.fishbournepreschool.org.uk/]]]

There should be a better messaging if the check returns no instances.

[peterwebster]

@anjackson @kinmanli just to confirm that it would be good if this endpoint was the QA Wayback soonish, so I can test the workflows

[peterwebster]

@anjackson @kinmanli is it going to be possible to see the daily check in action before we go into production ? I imagine it might not be straightforward, but I don't think it is running so far - I would expect lots of instances to show up at http://www.webarchive.org.uk/actdev/qa/list?f=bbc.co.uk if it was

[kinmanli]

QA - http://crawler03.bl.uk:8080/wayback/xmlquery.jsp?url=http://www.bbc.co.uk/

[kinmanli]

Friendlier message when there are no new instances or bad url. Also, set to QA wayback url

[anjackson]

@kinmanli to answer @peterwebster's question, is there a hook URL that will look for new instances for all targets?

[kinmanli]

@anjackson @peterwebster I've created a script to do it (was thinking CRON) https://github.com/ukwa/w3act/blob/schemarefactor/waybackinstance_import.sh which calls a class

Can easily attach it to a URL?

[anjackson]

Oh, that's fine then. I was suggesting a web hook because I thought that might be easier, but you've already written a script so that's simpler to cron. Let's ask @GilHoggarth if he'll add an act-dev hook that will run your script nightly...

[GilHoggarth]

So, I should configure this 'waybackinstance_import.sh' script to run nightly. @kinmanli Given /actdev redeploys/restarts daily at 17, when should i schedule this script to run?

[kinmanli]

@GilHoggarth depends when @peterwebster will want to start testing this functionality?

[peterwebster]

@GilHoggarth @kinmanli well, I guess that out of normal hours is the best time, so this evening, after the service has redeployed ? No preference as to exact time

[GilHoggarth]

@kinmanli Looking at the actual 'waybackinstance_import.sh' script, I'm afraid that that won't work. The PLAY_HOME envar won't resolve as this script lives as /opt/w3act/ - so3 levels up is actually to /, and I don't have a /tools/ directory for play 2.2.1 (even though, to clarify, we are using v2.2.1). I'm assuming therefore that your dev environment is different from the server environment.

[kinmanli]

@GilHoggarth fixed (needed to run import)

[GilHoggarth]

/actdev is a "play target" service so it's libraries are scattered whereas w3act is a "play dist" service with all the libraries in one directory. So the triggering script (that runs the actual script) is slightly different on /actdev compared to w3act. Anyhoo, this is scheduled on both servers as a late night crontab, and both are being run at the moment to test that they work as expected.

[GilHoggarth]

/actdev script was referencing the prod.conf not application.conf, so changed this and rerunning. If run completes as expected, I'll close this ticket.

[GilHoggarth]

Import on /actdev worked as expected. Closing ticket.

[GilHoggarth]

Reopening so the same test is performed on w3act.

[GilHoggarth]

Ran the waybackinstance_import.sh script on w3act - it logged its progress and claimed "finished". @kinmanli As you know what this does, can you check that it has done what it should do via the UI please.

[kinmanli]

@GilHoggarth if you can make my user an archivist then I'll be able to see the results.

I've just checked the old listing code for instances and the results are only viewable by sysadm and archivist. Is this correct or should it be viewable by all roles @peterwebster?

[peterwebster]

@kinmanli at which point in the UI is the view you're talking about ?

[kinmanli]

@peterwebster

https://www.webarchive.org.uk/act/instances/listbytarget/?t=790
https://www.webarchive.org.uk/act/instances/list/

[GilHoggarth]

@kinmanli Done, you're now an w3act archivist.

[kinmanli]

@GilHoggarth this looks good with the latest ones being 24/02/2015

There are 3 from 25/02/2015 on http://www.webarchive.org.uk/actdev/wayback/*/http://www.theguardian.com/ but I guess they will get imported on the next run

[anjackson]

I think instances should be visible to everyone, surely?

[nicolabingham]

Yes, as users at all levels will want to see what has been crawled and when.

[GilHoggarth]

@kinmanli Following Nicola's above comment I've tested https://www.webarchive.org.uk/act/instances/listbytarget/?t=790 to be accessible by the admin user and by me (gil.hoggarth, just user!) I couldn't see the results as me

[kinmanli]

@GilHoggarth @nicolabingham my user with role 'viewer' can now view and search instances.

checked in

Also, I removed all trailing slashes '/' (after 'listbytarget'). So:

https://www.webarchive.org.uk/act/instances/listbytarget/?t=790

Is now:

https://www.webarchive.org.uk/act/instances/listbytarget?t=790

Same for:

https://www.webarchive.org.uk/act/instances/list?f=http%3A%2F%2Fwww.islamic-relief.or

[GilHoggarth]

Note To be tested after new w3act deployment

[GilHoggarth]

https://www.webarchive.org.uk/act/instances/listbytarget?t=790 can now be viewed by a 'user' account, the admin account, but not by a not-logged-in visitor.
Running import script again to check new data still comes in...

[GilHoggarth]

Script run completed and new imports seen - closing ticket.