WACZ Aggregation / Multi WACZ Specification

[edsu]

Details about how to aggregate multiple WACZ files into a single WACZ need to be added to the specification. This hinges on resources in the datapackage.json using a url for a WACZ rather than a path. See the Resource Information section in the Data Package specification for details:

{
   "resources": [
      {"hash": "...", "url": "https://example.com/filename_1.wacz", "bytes": "..."}
      {"hash": "...", "url": "https://example.com/filename_2.wacz", "bytes": "..."}
   ]
   ...
}

There should also be a Data Package profile so that clients can easily distinguish between collections and regular WACZ files. Perhaps WACZ-Aggregation?

The specification should document that WACZ users MAY want to use the data-package.json as a place to record additional metadata about crawls. See the browsertrix-cloud API for examples.

[ikreymer]

I wonder if this should be a separate specification, to avoid confusion. It is no longer a WACZ, which is a specific format, but a collection of WACZ, probably would be just a .json file (the datapackage.json) to start, which is a different specification, but of course depends on the WACZ.

The collection spec is sort of independent from the file format spec imo.

[edsu]

I don't think it should be separate. Unless I am misunderstanding (which is very possible) an understanding of a WACZ Aggregation without an understanding of WACZ would be pretty much useless. Why make it more difficult to manage as two separate specifications?

[ikreymer]

I don't think it should be separate. Unless I am misunderstanding (which is very possible) an understanding of a WACZ Collection without an understanding of WACZ would be pretty much useless. Why make it more difficult to manage as two separate specifications?

Well, it's really a completely different format, an aggregate format of wacz (and possibly other types) to form a collection.
The only overlap is that currently thinking of this also as a frictionless data package (though maybe that doesn't make sense given that it uses a url instead of an internal path).

Maybe it should really be called a 'web archive collection', which consists of collection-level data.

For example, maybe a collection could have a mix of wacz and regular warc files, which could be both be listed in the resources section. As conceived now, it's just a single json file, though could also change.

[ikreymer]

Another aspect that the 'WACZ aggregate' could have is a page-to-wacz mapping, along with resources, for example:

{
  "resources": [{
     "name": "example-com-crawl",
     "url": "https://store.example.org/example-com-crawl.wacz",
     "hash": ...
     "bytes": ...
   }, {
      "name": "another-site-com-crawl",
     "url": "https://store.example.org/another-site-com-crawl.wacz",
     "hash": ...
     "bytes": ...
    }],
    
   "pages": [
      "https://example.com/": {"filename": "example-com-crawl"},
      "https://another-site.example.com": {"filename":  "another-site-com-crawl"}
    ]
  }

This would help route the page to the correct wacz file, otherwise, would need to search through all of them..

[edsu]

I like WACZ Aggregation better than WACZ Collection, and have just updated some of the text above.

[edsu]

I guess my point is that if someone is building a WACZ client, as we want people to do, they will want to be clear about what their viewer needs to do. I think the easiest way to communicate that is with a single specification about what WACZ support means. If implementors need to digest multiple specifications I think we will risk losing them.

[DiegoPino]

Hi. Maybe something like a iiif collection manifest (so just a json or json-ld) might make sense here?

On Wed, Mar 9, 2022 at 12:19 AM Ilya Kreymer ***@***.***> wrote: I don't think it should be separate. Unless I am misunderstanding (which is very possible) an understanding of a WACZ Collection without an understanding of WACZ would be pretty much useless. Why make it more difficult to manage as two separate specifications? Well, it's really a completely different format, an aggregate format of wacz (and possibly other types) to form a collection. The only overlap is that currently thinking of this also as a frictionless data package (though maybe that doesn't make sense given that it uses a url instead of an internal path). Maybe it should really be called a 'web archive collection', which consists of collection-level data. For example, maybe a collection could have a mix of wacz and regular warc files, which could be both be listed in the resources section. As conceived now, it's just a single json file, though could also change. — Reply to this email directly, view it on GitHub <https://github.com/webrecorder/wacz-spec/issues/112#issuecomment-1062514371>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABU7ZZ5VHUN5EOD3PCCAGPLU7AKDTANCNFSM5QCZLZ5A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Diego Pino Navarro Digital Repositories Developer Metropolitan New York Library Council (METRO)

[ikreymer]

Hi. Maybe something like a iiif collection manifest (so just a json or json-ld) might make sense here?

Hm, that's an interesting idea.. yeah, its basically a collection manifest for multiple WACZs (and possibly WARCs), maybe that's the best way to look at it..

[edsu]

That's a useful comparison @DiegoPino!

It might be helpful to think about the WACZ specification as the equivalent of the Image API and this Aggregated WACZ view as the equivalent of the Presentation API. I could even imagine expressing the Aggregated WACZ as a IIIF Manifest since the IIIF Presentation API is oriented around the abstract idea of Canvases that can include images, video, audio. Why not a web archive too? But I fear that this might be a bit of a leaky abstraction because the sequencing of WACZs doesn't make a lot of sense in this context? Also the WACZ itself contains a lot of presentation metadata itself.

While the separation between Image and Presentation APIs in IIIF allows it to be more general I also think it was because they were developed separately in time. As someone who had to implement support for them at one point I can say that understanding and tracking them as separate specifications sometimes proved challenging. But I suspect other people may feel differently about that.

I think what you've identified here is that there's a bit of slippage between the WACZ media type (the ZIP file on the web) and WACZ as an API, which we see developing in tools like Browsertrix Cloud and which relates to other work like WASAPI. I also think this crops up in the Unzipped WACZ use case...

[edsu]

I noticed an inconsistency in the Frictionless DataPackage specification around URLs in Resources.

The Data Resource specification says:

path MUST be a string – or an array of strings (see “Data in Multiple
Files”). Each string MUST be a “url-or-path” as defined in the next section.

whereas another Data Package specification distinguishes between url and path. I think we should use the former (url-or-path) since it maps to implementations we have already, and the url usage seems to be from an earlier version of the DataPackage specification.