Add support for the new fetcher cache in Mesos 0.23

[ssk2]

It looks like this requires additional configuration and for the fetcher cache boolean to set to be true when tasks are executed (as per https://github.com/apache/mesos/blob/master/docs/fetcher.md). It would be nifty if Marathon supported this feature!

This is related to #1377.

[pdiorio]

+1 This would be a very useful addition!

[mindscratch]

👍

[sybrandy]

idea++ 👍

[mindscratch]

Any chance this will make it in the 0.13.0 release?

[mindscratch]

I guess not. Perhaps 0.14?

[wendigo]

I can work on this but I need a tip from core-devs how to expose it via API :)

Looking at source in V2AppDefinition: uris: Seq[String] = AppDefinition.DefaultUris, this cannot be extended to be backwards compatible. We can add another field that will be combined with uris though.

[wendigo]

@aquamatthias looking forward for your opinion on that :)

[aquamatthias]

The URI's in mesos look like this:

 message URI {
    required string value = 1;
    optional bool executable = 2;
    optional bool extract = 3 [default = true];
    optional bool cache = 4;
  }

In order to support this data structure, we have to move away from simple strings.
The new field should allow expressing those properties, where all properties than the URI are optional. Example:

[
  { "uri": "http://example.com/bla" },
  { "uri": "http://example.com/foo", "executable": true, "extract": false, "cache": false }
]

To make this backward compatible, the uris property should stay as is with defaults applied.

Since uris is a pretty good name, we need another one.
What would be a good name?

[wendigo]

Maybe something like "artifacts" since we are downloading them in order to launch our application.

[wendigo]

Or "fetchUris" or "downloadUris".

[mindscratch]

I'd prefer uris change to new format. As you mentioned, today there are default values being used for the other properties, why not expose them? There have been backward incompatible changes made before.

[wendigo]

@mindscratch I do not like breaking compatibility if it can be avoided.

[aquamatthias]

We already also have storeUrls - which makes it even more confusing...
At least in the format we should be backward compatible.

[mindscratch]

So what would be the difference between uris and say, fetchUris? It seems confusing to have two different properties that might do the same thing.

[wendigo]

I think that best solution is to provide V3 API in which every primitve type (i.e. string) is replaced with class enclosing this type (so "uris": ["uri1"] becomes "uris": [{"uri": "uri1"}]). Since we can't do that I would introduce new field and mark old one as deprecated to be dropped in next versions.

[wendigo]

Any thoughts @aquamatthias ?

[aquamatthias]

Hey @wendigo had the same idea in mind (see comment above) - introduce a new parameter with new format which supersedes 'uris'.

[wendigo]

So let's settle on naming and I can implement this feature :)

[mindscratch]

I'll vote for fetchUris ...can't wait to see this land.
On Dec 27, 2015 1:35 PM, "Mateusz Gajewski" notifications@github.com
wrote:

So let's settle on naming and I can implement this feature :)

—
Reply to this email directly or view it on GitHub
#1953 (comment)
.

[aquamatthias]

Naming is hard ...
What about the simple version?
fetch:[{uri:xxx}]

Am 27.12.2015 um 22:10 schrieb Craig Wickesser notifications@github.com:

I'll vote for fetchUris ...can't wait to see this land.
On Dec 27, 2015 1:35 PM, "Mateusz Gajewski" notifications@github.com
wrote:

So let's settle on naming and I can implement this feature :)

—
Reply to this email directly or view it on GitHub
#1953 (comment)
.

—
Reply to this email directly or view it on GitHub.

[wendigo]

Ok, I'm on it :)

[mindscratch]

Sounds good
On Dec 28, 2015 11:13 AM, "Mateusz Gajewski" notifications@github.com
wrote:

Ok, I'm on it :)

—
Reply to this email directly or view it on GitHub
#1953 (comment)
.

[wendigo]

Almost there 😄

Tests: succeeded 849, failed 0, canceled 0, ignored 2, pending 0

[mindscratch]

Awesome!

[wendigo]

Done!