LoMap format specification

[gonzalo-rr]

This Issue is only to discuss the format to use to save the data in the SOLID pods (such as using plain JSON, using JSON-LD, following the RDF format, etc), not to discuss what information to store (latitude, longitude, description, score, etc).

Please stick to the subject, and use other issues to talk about other subjects, thank you.

[andrrsin]

I am uploading a small example given by Schema of how a place can look using JSON-LD. At the same time i link here the Place schema. It has more examples of JSON-LD with the different variables provided by the initiative. @iimxinn I tag you as you requested in the issue #4 to upload an example.

{
  "@context": "https://schema.org",
  "@type": "Place",
  "geo": {
    "@type": "GeoCoordinates",
    "latitude": "40.75",
    "longitude": "-73.98"
  },
  "name": "Empire State Building"
}

[gitblanc]

You should check this link to get further information about rdf -> https://arquisoft.github.io/viadeSpec/#definitions

[gonzalo-rr]

Hello everyone.
I was looking into the different ways of storing data in the SOLID pods to see the differences of using JSON, JSON-LD and RDF.

After some trying I figured out how to do it with in all of those ways. I just wanted to leave the code here so anyone can use it to store data and maybe understand better how data is stored in SOLID so we can make better decisions about the format we want to use.

I leave the code to store data using JSON-LD first because I think most people will be more interested in JSON-LD.

Sorry in advance for the length, you can just read the part that you are interested in, if you have any doubts about the code please ask, I know it's not great code but it is just an example.

export async function addPlace(session: Session) {
    try {
        let url = "https://gonzalo99.inrupt.net/public/test";

        let place: JsonLdDocument = {
            "@context": "https://schema.org/",
            "@type": "Place",
            "name": "Campo de San Francisco",
            "latitude": "43.361717",
            "longitude": "-5.850186"
        };

        let blob = new Blob([JSON.stringify(place)], { type: "application/ld+json" });
        let file = new File([blob], place.name + ".jsonld", { type: blob.type });

        await overwriteFile(
            url,
            file,
            { contentType: file.type, fetch: session.fetch }
        );
    } catch (error) {
        console.log(error);
    }
}

And here is the result in my POD:

Please note that I'm using the functions provided by the inrupt solid-client library. I'm also using the overwriteFile function, which may not be the best function to use in this case, but I used it to give a simpler example.

Now to give you an example of writing data into a SOLID POD using Things and Datasets:

export async function addPlace(session: Session) {
    try {
        let dataset = createSolidDataset();
        await saveSolidDatasetAt(
            "https://gonzalo99.inrupt.net/public/test",
            dataset,
            { fetch: session.fetch }
        );
    } catch (error) {
        console.log(error);
    }

    try {
        let url = "https://gonzalo99.inrupt.net/public/test";
        let dataset = await getSolidDataset(
            url,
            { fetch: session.fetch }
        );
        let place = buildThing(createThing({name: "sanfran"}))
            .addStringNoLocale(SCHEMA_INRUPT.name, "Campo de San Francisco")
            .addStringNoLocale(SCHEMA_INRUPT.latitude, "43.361717")
            .addStringNoLocale(SCHEMA_INRUPT.longitude, "-5.850186")
            .addUrl(RDF.type, "https://schema.org/Place")
            .build();
        dataset = setThing(dataset, place);
        await saveSolidDatasetAt(
            url,
            dataset,
            { fetch: session.fetch }
        );
    } catch (error) {
        console.log(error);
    }
}

And this is the result:

The result is identical to using JSON-LD. If you want more information here is where I got the information.

Finally, this is an example of using JSON to store data (I leave it so you can compare the three, but most of you probably have similar functions):

export async function addPlace(session: Session) {
    try {
        let url = "https://gonzalo99.inrupt.net/public/test";

        let place: JsonLdDocument = {
            "@context": "https://schema.org/",
            "@type": "Place",
            "name": "Campo de San Francisco",
            "latitude": "43.361717",
            "longitude": "-5.850186"
        };

        let blob = new Blob([JSON.stringify(place)], { type: "application/json" });
        let file = new File([blob], place.name + ".json", { type: blob.type });

        await overwriteFile(
            url,
            file,
            { contentType: file.type, fetch: session.fetch }
        );
    } catch (error) {
        console.log(error);
    }
}

Result:

The only difference between JSON-LD and JSON is the mime type.

[gitblanc]

Hello everyone. I was looking into the different ways of storing data in the SOLID pods to see the differences of using JSON, JSON-LD and RDF.

After some trying I figured out how to do it with in all of those ways. I just wanted to leave the code here so anyone can use it to store data and maybe understand better how data is stored in SOLID so we can make better decisions about the format we want to use.

I leave the code to store data using JSON-LD first because I think most people will be more interested in JSON-LD.

Sorry in advance for the length, you can just read the part that you are interested in, if you have any doubts about the code please ask, I know it's not great code but it is just an example.

export async function addPlace(session: Session) {
    try {
        let url = "https://gonzalo99.inrupt.net/public/test";

        let place: JsonLdDocument = {
            "@context": "https://schema.org/",
            "@type": "Place",
            "name": "Campo de San Francisco",
            "latitude": "43.361717",
            "longitude": "-5.850186"
        };

        let blob = new Blob([JSON.stringify(place)], { type: "application/ld+json" });
        let file = new File([blob], place.name + ".jsonld", { type: blob.type });

        await overwriteFile(
            url,
            file,
            { contentType: file.type, fetch: session.fetch }
        );
    } catch (error) {
        console.log(error);
    }
}

And here is the result in my POD:

Please note that I'm using the functions provided by the inrupt solid-client library. I'm also using the overwriteFile function, which may not be the best function to use in this case, but I used it to give a simpler example.

Now to give you an example of writing data into a SOLID POD using Things and Datasets:

export async function addPlace(session: Session) {
    try {
        let dataset = createSolidDataset();
        await saveSolidDatasetAt(
            "https://gonzalo99.inrupt.net/public/test",
            dataset,
            { fetch: session.fetch }
        );
    } catch (error) {
        console.log(error);
    }

    try {
        let url = "https://gonzalo99.inrupt.net/public/test";
        let dataset = await getSolidDataset(
            url,
            { fetch: session.fetch }
        );
        let place = buildThing(createThing({name: "sanfran"}))
            .addStringNoLocale(SCHEMA_INRUPT.name, "Campo de San Francisco")
            .addStringNoLocale(SCHEMA_INRUPT.latitude, "43.361717")
            .addStringNoLocale(SCHEMA_INRUPT.longitude, "-5.850186")
            .addUrl(RDF.type, "https://schema.org/Place")
            .build();
        dataset = setThing(dataset, place);
        await saveSolidDatasetAt(
            url,
            dataset,
            { fetch: session.fetch }
        );
    } catch (error) {
        console.log(error);
    }
}

And this is the result:

The result is identical to using JSON-LD. If you want more information here is where I got the information.

Finally, this is an example of using JSON to store data (I leave it so you can compare the three, but most of you probably have similar functions):

export async function addPlace(session: Session) {
    try {
        let url = "https://gonzalo99.inrupt.net/public/test";

        let place: JsonLdDocument = {
            "@context": "https://schema.org/",
            "@type": "Place",
            "name": "Campo de San Francisco",
            "latitude": "43.361717",
            "longitude": "-5.850186"
        };

        let blob = new Blob([JSON.stringify(place)], { type: "application/json" });
        let file = new File([blob], place.name + ".json", { type: blob.type });

        await overwriteFile(
            url,
            file,
            { contentType: file.type, fetch: session.fetch }
        );
    } catch (error) {
        console.log(error);
    }
}

Result:

The only difference between JSON-LD and JSON is the mime type.

Nice code!

But using this way to store JSON-ld (that needs to only modify the actual JSON spec adding @context and @type, very similar by the way) gives me several questions:

• Do you need to add a new file when you add a new place or is it stored in the same file?
• How do you extract the current info that you stored? Using FOAF.property or it is similar to JSON => file.map[x].location[x].name?
• How do you contemplate the multiple map option?
• How do you contemplate the collection of comments and reviewScores?

[gonzalo-rr]

Good morning, those are really good questions. I have been trying to find an answer to all of them, however I do not have a perfect answer to all.

I'll try to give an answer to each one:

• Adding a new place could be done by retrieving the map file, adding a new place to the list of places and finally overwritting the existing map file. It may not be the most elegant way but I do not see another one.
• I have tried extracting the stored information using the tools provided by SOLID to read structured data (more info here), however I could not get it to work. It can definitely be done by getting the file as a JSON, like reading non structured data (more info here). Here's an example of a function to do that.

export async function getPlace(session: Session) {
    try {
        let url = "https://gonzalo99.inrupt.net/public/test/sanfran";

        let blob = await getFile(
            url,
            { fetch: session.fetch }
        );

        let place = JSON.parse(await blob.text());

        console.log(place);
    } catch (error) {
        console.log(error);
    }
}

And here's the result:

• About the multiple map option, each map coud be a Map, each one stored in a different JSON-LD file, and each place could be a Place contained in a property of each Map. The Map schema has various properties (I don't know if properties is the right word) that are a Place or a list of Place such as contentLocation, locationCreated, spatial and spatialCoverage. I think this last one may be the best option.
  Here's an example in code:

export async function addMap(session: Session) {
    try {
        let url = "https://gonzalo99.inrupt.net/public/test/map";

        let sanfran: JsonLdDocument = {
            "@context": "https://schema.org/",
            "@type": "Place",
            "name": "Campo de San Francisco",
            "latitude": "43.361717",
            "longitude": "-5.850186"
        };

        let catedral_oviedo: JsonLdDocument = {
            "@context": "https://schema.org/",
            "@type": "Place",
            "name": "Santa Iglesia Basílica Catedral Metropolitana de San Salvador de Oviedo",
            "latitude": "43.362583",
            "longitude": "-5.843081"
        };

        let mapa: JsonLdDocument = {
            "@context": "https://schema.org/",
            "@type": "Map",
            "name": "Mapa Oviedo",
            "spatialCoverage": [sanfran, catedral_oviedo]
        }

        let blob = new Blob([JSON.stringify(mapa)], { type: "application/ld+json" });
        let file = new File([blob], mapa.name + ".jsonld", { type: blob.type });

        await overwriteFile(
            url,
            file,
            { contentType: file.type, fetch: session.fetch }
        );
    } catch (error) {
        console.log(error);
    }
}

And here's the result:

• As for the reviews and comments, the aggregateRating property is a AggregateRating that exists in both in Map and in Place. It makes more sense using it in Place, but I think it is just a number (or maybe numbers) with no property for comments. I'll have to look into it more in the future, right now I don't know how to use it.

[gonzalo-rr]

By the way, if you want to run the code I provided, please remember to use this import statement:

import {JsonLdDocument} from "jsonld/jsonld";

[gitblanc]

It seems nice! However, I've been loading markers to the map from the JSON spec and I don't know if with this specification will happen the same, but extracting data from pods is too slow (maybe 4 secs if you have more than 100 markers). Can you let me know how much does it least when you have this spec implemented in your code?

[gonzalo-rr]

I will let you know when I implmement it fully.

[gonzalo-rr]

It seems nice! However, I've been loading markers to the map from the JSON spec and I don't know if with this specification will happen the same, but extracting data from pods is too slow (maybe 4 secs if you have more than 100 markers). Can you let me know how much does it least when you have this spec implemented in your code?

I finally implemented it, I didn't time it but with aprox. 100 points it's something like a second.

[Omitg24]

Good night!
I've created a format structure in our wiki, you can access through this link.

Please let me know your thoughts and feel free to suggest any improvements you may have! I've tried to follow our previous structure (with the modifications we have commented throughout this weeks). I have also included Schema Types, such as Review, ImageObject, Rating, etc...

[DavidGonzalezFernandez]

As proposed in Data specification with format JSON LD a problem arises when allowing third users to add scores, comments, and photos to a user's places. Let us consider the example of Alice creating a place and Bob adding a comment to Alice's place. The issues are outlined below.

Trustworthiness:

If Bob's comments are stored in Alice's POD, she could:

• Modify the text of Bob's comment.
• Modify Bob's user ID to make it appear that Charlie wrote that comment.
• Create new comments and associate them with Charlie (who is Charlie?, that's exactly the point).

Therefore, we face a problem: just by looking at the file's contents we cannot be certain that the information added by third parties is correct.

Ownership of information:

The SOLID project's philosophy is for each user to store their data in their POD. Our specification goes against data decentralization and self-governance of one's data, since Bob's comments are stored in Alice's POD.

Permissions:

We want Bob to be able to add comments to Alice's places. Given our specification:

• If Bob only has read permission for Alice's resources, he cannot add his comment.
• If Bob has append permission for Alice's resources, he can add his comment but cannot modify or delete it later. Potentially, he could also create comments and assign them to Charlie.
• If Bob has write permission for Alice's resources, he can add, modify, and delete his comments. He could also modify the rest of the information contained in the resources, further exacerbating the trustworthiness problem.

---

While the examples reference comments, this also applies to scores, photographs, and other information from third parties that we allow to add to a user's places.

It's important to emphasize that the current highlighted problem does not automatically indicate that the entire specification is flawed or unusable. Rather, the specification is still a work in progress and there is room for improvement in future versions. Our focus should be on identifying and addressing the problem while continuing to work towards refining and optimizing the specification. We shouldn't abandon the effort entirely, but rather keep moving forward and striving for improvement.

[Omitg24]

About security problems

I agree with you that security @DavidGonzalezFernandez could be a serious issue, but at the moment, I don't have a perfect solution for how to address it, I'd wait to hear what other people propose on how to sort it out.

New format structure about coordinates

Regarding the coordinates section, I made a few changes. Instead of using the "geo" field, which might be unnecessary since "Place" already includes latitude and longitude, I simplified it to:

"latitude": "43.35302550278352",
"longitude": "-5.849442050492078",

I believe this new format is more straightforward and easier to understand

[gonzalo-rr]

As proposed in Data specification with format JSON LD a problem arises when allowing third users to add scores, comments, and photos to a user's places. Let us consider the example of Alice creating a place and Bob adding a comment to Alice's place. The issues are outlined below.

Trustworthiness:

If Bob's comments are stored in Alice's POD, she could:

• Modify the text of Bob's comment.
• Modify Bob's user ID to make it appear that Charlie wrote that comment.
• Create new comments and associate them with Charlie (who is Charlie?, that's exactly the point).

Therefore, we face a problem: just by looking at the file's contents we cannot be certain that the information added by third parties is correct.

Ownership of information:

The SOLID project's philosophy is for each user to store their data in their POD. Our specification goes against data decentralization and self-governance of one's data, since Bob's comments are stored in Alice's POD.

Permissions:

We want Bob to be able to add comments to Alice's places. Given our specification:

• If Bob only has read permission for Alice's resources, he cannot add his comment.
• If Bob has append permission for Alice's resources, he can add his comment but cannot modify or delete it later. Potentially, he could also create comments and assign them to Charlie.
• If Bob has write permission for Alice's resources, he can add, modify, and delete his comments. He could also modify the rest of the information contained in the resources, further exacerbating the trustworthiness problem.

While the examples reference comments, this also applies to scores, photographs, and other information from third parties that we allow to add to a user's places.

It's important to emphasize that the current highlighted problem does not automatically indicate that the entire specification is flawed or unusable. Rather, the specification is still a work in progress and there is room for improvement in future versions. Our focus should be on identifying and addressing the problem while continuing to work towards refining and optimizing the specification. We shouldn't abandon the effort entirely, but rather keep moving forward and striving for improvement.

I think you are absolutely right, those are some really interesting points.

Those problems may be hard to solve so I was wondering if you, @DavidGonzalezFernandez, had any idea of how we could approach solving them.

I personally don't know how it could be solved without using some type of cryptography, but that may take too long and it may be too difficult to implement in time.

[DavidGonzalezFernandez]

I have been pondering this problem for several days and have come to the following solution:

Alice creates a place:

• Alice's place is stored in Alice's POD.

Bob wants to add a comment to Alice's place:

• A new line is added to Alice's file (Bob only needs append permission) with the necessary information to access the comment:
  • The user's webID
  • The comment's ID
• The comment is saved in a file in Bob's POD.

In this way:

• Alice cannot modify Bob's comment (she does not have permission to modify Bob's file).
• Alice cannot create false comments and assign them to Charlie (the files in Charlie's POD would not exist).
• Bob's data is stored in Bob's POD (Bob owns his data)
• Bob can modify his comment (he just needs to modify his own file).
• Bob can delete his comment (he just needs to delete his own file).
• Bob does not have permission to modify the rest of the place information (he only has permission to append).

The remaining problems are:

• Alice would have permission to delete the information linking to Bob's comment (by deleting the only reference to that comment, the comment itself would not be deleted, but there would be no way to know that said comment belongs to that particular place).
• Alice could add references to existing real comments (but intended for a different place B) and assign them to place A.

The solution is somewhat more complex than the last proposal, but conceptually I believe it solves all the core problems I raised before. It still needs to be implemented and the feasibility of the proposed solution verified. I will notify you @gonzalo-rr @Omitg24 when it is completely implemented.