Curl Interactions

Henry Story edited this page Oct 21, 2015 · 15 revisions

This page describes a full interaction scenario with rww-play using curl. WARNING: FOR DEVELOPERS only.

curl is a very important tool for developers working on web projects. It is a very lightweight tool for HTTP interactions that allows one to test requests on the web. It is very useful for bug reports, as it makes it easy to set up situations to reproduce bugs without requiring a full browser installation. Since the point of rww-play is that the web is a peer to peer system, where servers can turn into clients and make requests to other servers, we don't want to presuppose a browser installation.

Some issues we have found are:

  • the HTTPS connections don't work in WANT mode: curl does not send a certificate if requested politely by the server. At this point it is different from many browsers.
  • OSX Mavericks bug: the --cert flag for curl no longer works and you will need to compile your own for reasons explained in this bug report. You can do this by using the Fink Project or Brew.

Basic Linked Data

The data in rww-play is stored as files on the file system, just as with the main Apache server. The example test_www directory contains the following files:

$ cd test_www/

$ ls -al
total 40
drwxr-xr-x   2 hjs  staff   238  1 Dec 00:47 .
drwxr-xr-x  19 hjs  staff  1054  1 Dec 00:03 ..
-rw-r--r--   1 hjs  staff   722 23 Nov 21:35 .acl.ttl
-rw-r--r--   1 hjs  staff    82 23 Nov 21:35 .ttl
lrwxr-xr-x   1 hjs  staff     8 23 Nov 21:35 card -> card.ttl
-rw-r--r--   1 hjs  staff   186 28 Nov 13:53 card.acl.ttl
-rw-r--r--   1 hjs  staff   922  1 Dec 00:42 card.ttl

card here is a symbolic link to a default Turtle representation. Once the server is started ( see the front page, that file can be fetched using curl with the following curl command: ( the -i so that we can see the headers returned, and the -k is there so that we don't have trouble with the localhost non certified certificate our server is using on the https port 8443)

$ curl -i -k https://localhost:8443/2013/card
HTTP/1.1 200 OK
Accept-Patch: application/sparql-update
Access-Control-Allow-Origin: *
Content-Type: text/turtle
ETag: "1417390950000|Success(922)"
Last-Modified: Sun, 30 Nov 2014 23:42:30 GMT
Link: <>; rel=type, <card.acl>; rel=acl
Content-Length: 978

<#me> <> <> .

_:node1981hgchsx1 a <> ;
	<> "b7cb16af0aeec58a4c0c05e0504a334382a1db7a8a092057f97c271439f7ff8cfd469b615934fa401b4b320b756cf017e16c8ee0d5afceed1a54390738720c67813b765e1bf9e310809e133b7f7c2aca34e185c3bdcd42fc40d84772ad691f36b9078c8e0079f64089ae0adcaa80d4186cf683403d6485e578dbde161a82b4e34650cb77fd274fe84bb7ae488a3236f146178cf836cc701b1d3c40c0d7a8e838afc209e3b5c825fa9702017b52492f4cf4bdeb089726e2778eb63b8854c8b366b2c5425f5dec236c02c8e760b7303adfb2a94bf835c2e28901abeca292d7ca04c1ae3c377e2d2f3e011be76868d941678a18c2abf78f98f796f493f2a946cf2d"^^<> ;
	<> 65537 .

<#me> <> _:node1981hgchsx1 ;
	<> "Me (StampleIO test webid)" ;
	a <> .

The server knows how to do content-negotiation, and can also serve the file in other formats such as Json-Ld. We do this by telling the server that we accept that mime type in a header that we add with the -H "Accept: application/ld+json" parameter to curl

$  curl -i -H "Accept: application/ld+json" -k https://localhost:8443/2013/card
HTTP/1.1 200 OK
Accept-Patch: application/sparql-update
Access-Control-Allow-Origin: *
Content-Type: application/ld+json
ETag: "1417390950000|Success(922)"
Last-Modified: Sun, 30 Nov 2014 23:42:30 GMT
Link: <>; rel=type, <card.acl>; rel=acl
Content-Length: 1245

  "@graph" : [ {
    "@id" : "#me",
    "@type" : "",
    "" : {
      "@id" : "_:node1981hgchsx1"
    "" : {
      "@id" : ""
    "" : "Me (StampleIO test webid)"
  }, {
    "@id" : "_:node1981hgchsx1",
    "@type" : "",
    "" : {
      "@type" : "",
      "@value" : "65537"
    "" : {
      "@type" : "",
      "@value" : "b7cb16af0aeec58a4c0c05e0504a334382a1db7a8a092057f97c271439f7ff8cfd469b615934fa401b4b320b756cf017e16c8ee0d5afceed1a54390738720c67813b765e1bf9e310809e133b7f7c2aca34e185c3bdcd42fc40d84772ad691f36b9078c8e0079f64089ae0adcaa80d4186cf683403d6485e578dbde161a82b4e34650cb77fd274fe84bb7ae488a3236f146178cf836cc701b1d3c40c0d7a8e838afc209e3b5c825fa9702017b52492f4cf4bdeb089726e2778eb63b8854c8b366b2c5425f5dec236c02c8e760b7303adfb2a94bf835c2e28901abeca292d7ca04c1ae3c377e2d2f3e011be76868d941678a18c2abf78f98f796f493f2a946cf2d"
  } ]

Those two representations are required by LDP, but we also provide others such as the older text/rdf+xml. Try it out!

Both of these representations determine exactly the same graph: They mean exactly the same thing, i.e. all worlds in which the one is true, the other is true too.

Web Access Control

At the top of the GET response on https://localhost:8443/2013/card we found the following Link header

Link: <>; rel=type, <card.acl>; rel=acl

This is specified by the Web Linking RFC 5988. It described the resource as having two relations. Each is separated by the comma , character, which in HTTP headers is counterintuitively weaker than the semicolon ; . The two relations are when expressed fully

  • <> type <> : the resource we got is an ldp:Resource
  • <> acl <card.acl>: the Acl for this resource is to be found at the relative url card.acl

The second relation is saying then that card.acl is the Web Access Control graph for that resource.

Each resource in rww-play has an attached Access Control Resource which it points to as shown by this diagram Web ACL example diagram

The content of the ACL is to be found on the file system

$ cat card.acl.ttl 
@prefix acl: <> . 

[] acl:accessTo <card>, <card.ttl> ;
	acl:agentClass <> ;
	acl:mode acl:Read .

<> acl:include <.acl> .

The acl for card just includes the acl for the directory/collection . (TODO: wac:include has not yet been defined in the Web Access Control Ontology)

$ cat card.acl.ttl 
@prefix wac: <> .
@prefix foaf: <> .

<> wac:include <.acl> .

You can also get the same file with curl with the command

$ curl -i -k https://localhost:8443/2013/card.acl

That file includes all the acts from the directory acl. And that says

$ cat .acl.ttl 
@prefix acl: <> . 
@prefix foaf: <> . 

[] acl:accessToClass [ acl:regex "https://(\\w+\\.)?localhost:8443/.*[.]acl" ];
   acl:mode acl:Read;
   acl:agentClass foaf:Agent .

[] acl:accessToClass [ acl:regex "https://(\\w+\\.)?localhost:8443/.*" ];
   acl:mode acl:Write, acl:Read;
   acl:agent <card#me> .

This says that all files on this server that refer to this ACL in their headers via the ACL relation are readable by anyone ( foaf:Agent ), and writeable by the Agent identified by the WebID whose relative uri is <card#me> i.e., since it is taken from the resource https://localhost:8443/2013/ by following the URI specification on relative URI resolution it is the agent referred to by <https://localhost:8443/2013/card#me> ( for more details see WebID Specification

So this is a quite usual setup: all the files on a web server a readable by everyone but only writeable by the administrator.

Updating a resource with PATCH

Let's try that out. What happens when we try to PATCH the file. Say we want to change the name from the current "Me (StampleIO test webid)" to say "Henry Story". In the example directory we have a SPARQL Updated example

$ cat ../eg/card.update
PREFIX foaf: <>
DELETE DATA { <https://localhost:8443/2013/card#me> foaf:name "Me (StampleIO test webid)" . };

PREFIX foaf: <>
INSERT DATA { <https://localhost:8443/2013/card#me> foaf:name "Henry Story" . }

( I find it a bit odd that in SPARQL Update one has to write the PREFIX out twice, but you can submit comments for improvements )

If we try to use the HTTP PATCH verb that the Linked Data Platform suggests we use then we get the following:

$ curl -X PATCH -k -i --data-binary @../eg/card.update \
    -H "Content-Type: application/sparql-update; utf-8"  \
HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=utf-8
WWW-Authenticate: Signature realm="/"
Content-Length: 3878

You are asked to retry using http-signatures authentication. This is implemented in very few tools at the moment, so it is easier to use client certificate authentication.

To try using the client certificate with curl we need to specify that we want to authorize with ClientCertificates like this:

$ curl -X PATCH -k -i --data-binary @../eg/card.update \
    -H "Content-Type: application/sparql-update; utf-8"  \
    -H "Authorization: ClientCertificate" \
curl: (56) SSL read: error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate, errno 0

But this gives us an ugly error. This is because curl does not allow WANT TLS connections, but only NEED, and so a failure to authenticate closes the connection. Most browsers have the more friendly behaviour allowing the server to return an HTTP error code and an explanatory body, as we saw above.

Requesting the same resource with the right client certificate we get a more interesting answer

$ curl -X PATCH -k -i --data-binary @../eg/card.update \
  -H "Content-Type: application/sparql-update; utf-8"   \
  -H "Authorization: ClientCertificate" \
  --cert ../eg/test-localhost.pem:test  https://localhost:8443/2013/card
HTTP/1.1 428 Precondition Required
Content-Type: text/plain; charset=utf-8
Content-Length: 38

altering the resource requires an ETag

The client certificate in ../eg/test-localhost.pem contains exactly the public key given in the above cert as you can see by comparing the modulus and exponents in both representations. This is what allows the authentication to go through, using the WebID over TLS protocol.

$ $ openssl x509 -in ../eg/test-localhost.pem -inform pem -text
        Version: 3 (0x2)
        Serial Number:
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: CN=WebID, O={}
            Not Before: Nov 27 15:07:38 2014 GMT
            Not After : Nov 24 15:17:38 2024 GMT
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: critical
            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment, Key Agreement, Certificate Sign
            X509v3 Basic Constraints: critical
            Netscape Cert Type: 
                SSL Client, S/MIME
    Signature Algorithm: sha1WithRSAEncryption

This server does not allow a patch without an If-Match header. ( this should be something configurable ). When we made our original request to the card it came with an tag

ETag: "1417390950000|Success(922)"

so we need to use this in our PATCH request

$ curl -X PATCH -k -i --data-binary @../eg/card.update \
    -H "Content-Type: application/sparql-update; utf-8" \
    -H 'If-Match: "1417390950000|Success(922)"'  \
    -H "Authorization: ClientCertificate" \
    --cert ../eg/test-localhost.pem:test  \
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
User: https://localhost:8443/2013/card#me
Content-Length: 9


And indeed it worked as shown by the 200 Ok response. The card should now have the new name.

Note that the original GET on the resource gave us the Allow header with the following values


and that this did not include PATCH, nor DELETE for the matter. That is because neither those methods are allowed by an anonymous user. An authenticated user would find those methods also listed. ( this is easier to verify in a browser )

Creating a data Resource

Next we can publish a couch surfing opportunity using HTTP's POST method as explained by the LDP spec

$ curl -X POST -k -i -H "Content-Type: text/turtle; utf-8"  \
    -H "Authorization: ClientCertificate" \
    --cert ../eg/test-localhost.pem:test  \
    -H "Slug: couch" -d @../eg/couch.ttl https://localhost:8443/2013/
HTTP/1.1 201 Created
Location: https://localhost:8443/2013/couch
Content-Length: 0

The Location: header in the above response tells us that the name of the created resource is https://localhost:8443/2013/couch.

We can now look at the contents of the https://localhost:8443/2013/ collection, where we should see - and do - the new couch resource listed as having been created by ldp.

$ curl -k -i -X GET \
   --cert ../eg/test-localhost.pem:test https://localhost:8443/2013/
HTTP/1.1 200 OK
Link: <https://localhost:8443/2013/.acl>; rel=acl
Content-Type: text/turtle
Content-Length: 701

<> <> <> ;
	<> <card> , <test/> , <couch> .

<test/> <> "1382465690000"^^<> ;
	a <> .

<couch> <> "1382465783000"^^<> ;
	<> "9"^^<> .

<card> <> "8"^^<> ;
	<> "1372357799000"^^<> .

Each resource in the LDPC is listed and the size of its Turtle representation on disk is given. Notice also that the LDPC </2013> is described as having been made by Henry Story. Such information is stored in the test_www/.ttl file, which can also be edited using the usual LDP protocol. (TODO: is this ok with LDP?)

We can find out about the ACL for this resource using HEAD (TODO: OPTIONS would be better, but is not implemented yet )

$ curl -X HEAD -k -i --cert ../eg/test-localhost.pem:test \
HTTP/1.1 200 OK
Link: <https://localhost:8443/2013/couch.acl>; rel=acl
Content-Type: text/turtle
Content-Length: 0

( TODO: The Content-Length should not be 0 for HEAD. Bug in Play2.0 probably )

So we add the couch acl which gives access to that information in addition to the default owner of the collection, to two groups of people by patching the acl with couch.acl.patch

$  curl -X PATCH -k -i -H "Content-Type: application/sparql-update; utf-8"  \
    -H "Authorization: ClientCertificate" \
    --cert ../eg/test-localhost.pem:test \
    --data-binary @../eg/couch.acl.patch \
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Content-Length: 9


This makes it available to the test user and the members of the WebID and OuiShare groups. If you have a WebID then try adding yours and test it. You can also request different formats by changing the Accept: header such as with the following request for RDF/XML

$ curl -k -i -X GET -H "Accept: application/rdf+xml" --cert ../eg/test-localhost.pem:test https://localhost:8443/2013/couch

or if you prefer rdf/xml over turtle as described by the Content Negotiation section of the HTTP1.1 spec:

$ curl -k -i -X GET -H "Accept:application/rdf+xml;q=0.9,text/turtle;q=0.7" --cert ../eg/test-localhost.pem:test https://localhost:8443/2013/couch

Creating a binary resource

You can also upload binary resources though at present the mime types uplowdable are artificially limited, and it just requires one to improve the SupportedBinaryMimeExtensions object.

Here is how you can upload that Apache feather logo to your web site:

curl -X POST -k -i -H "Content-Type: image/png" \
   -H "Authorization: ClientCertificate" \
   --cert ../eg/test-localhost.pem:test \
   -H "Slug: Apache-feather" \
   --data-binary @../eg/apache-feather.png \

SPARQL Queries

It is then possible to also use SPARQL queries on particular resources. (TODO: find a better example)

$ cat ../eg/couch.sparql 
PREFIX gr: <> 
  [] a <>;
     gr:description ?D .

$ curl -X SEARCH -k -i -H "Content-Type: application/sparql-query; charset=UTF-8" \
    -H "Authorization: ClientCertificate" \
    --cert ../eg/test-localhost.pem:test \
    --data-binary @../eg/couch.sparql https://localhost:8443/2013/couch
HTTP/1.1 200 OK
Content-Type: application/sparql-results+xml
Content-Length: 337

<?xml version='1.0' encoding='UTF-8'?>
<sparql xmlns=''>
        <variable name='D'/>
            <binding name='D'>
                <literal datatype=''>Comfortable couch in Artist Stables</literal>

You can also send a DESCRIBE query ( which sadly is not very well defined, so that it may not return the same with different SPARQL engines. This is what is returned with the sesame engine:

$ cat ../eg/card-describe.sparql 
DESCRIBE <https://localhost:8443/2013/card#me>
$ curl -X SEARCH -k -i \
    -H "Content-Type: application/sparql-query; charset=UTF-8" \
    -H "Authorization: ClientCertificate" \
    --cert ../eg/test-localhost.pem:test \
    --data-binary @../eg/card-describe.sparql \
HTTP/1.1 200 OK
Content-Type: text/turtle
Content-Length: 1121

<https://localhost:8443/2013/card#me> a <> ;
	<> "Me (StampleIO test webid)" ;
	<> <> ;
	<> _:node199fktmr7x1 .

_:node199fktmr7x1 a <> ;
	<> 65537 ;
	<> "b7cb16af0aeec58a4c0c05e0504a334382a1db7a8a092057f97c271439f7ff8cfd469b615934fa401b4b320b756cf017e16c8ee0d5afceed1a54390738720c67813b765e1bf9e310809e133b7f7c2aca34e185c3bdcd42fc40d84772ad691f36b9078c8e0079f64089ae0adcaa80d4186cf683403d6485e578dbde161a82b4e34650cb77fd274fe84bb7ae488a3236f146178cf836cc701b1d3c40c0d7a8e838afc209e3b5c825fa9702017b52492f4cf4bdeb089726e2778eb63b8854c8b366b2c5425f5dec236c02c8e760b7303adfb2a94bf835c2e28901abeca292d7ca04c1ae3c377e2d2f3e011be76868d941678a18c2abf78f98f796f493f2a946cf2d"^^<> .

<https://localhost:8443/2013/card> <> <https://localhost:8443/2013/card#me> .

Note how this returns the symmetric concise bounded description, that is the subgraph of the graph consisting of all arcs going from the and into original URI, and the arc going from those nodes if the nodes are blank nodes, until it reaches either literals or URIs.

DELETEing a resource

Finally if you no longer want the couch surfing opportunity to be published you can DELETE it. ( It would be better to express that it was sold: DELETEing resources on the Web is usually bad practice: it breaks the links that other people set up to your services )

curl -k -i -X DELETE -H "Accept: text/turtle" \
   -H "Authorization: ClientCertificate" \
   --cert ../eg/test-localhost.pem:test https://localhost:8443/2013/couch
HTTP/1.1 200 OK
Content-Length: 0

And so the resource no longer is listed in the LDPC

$ curl -k -i -X GET \
     --cert ../eg/test-localhost.pem:test \
     -H "Authorization: ClientCertificate" \
HTTP/1.1 200 OK
Link: <https://localhost:8443/2013/.acl>; rel=acl
Content-Type: text/turtle
Content-Length: 109

<> a <> ;
    <> <card> , <raw/> , <test/> .

Creating a Container

To create a new container one just creates an LDP Resource that contains the triple <> a ldp:Container.

$ cat ../eg/newContainer.ttl 
@prefix ldp: <> .
@prefix foaf: <> .

<> a ldp:Container;
   foaf:topic "A container for some type X of resources";
   foaf:maker <../card#me> .

$  curl -X POST -k -i -H "Content-Type: text/turtle; utf-8"  \
   -H 'Link: <>; rel="type"' \
   --cert ../eg/test-localhost.pem:test \
   -H "Slug: XDir" -d @../eg/newContainer.ttl \

HTTP/1.1 201 Created
Location: https://localhost:8443/2013/XDir/
Content-Length: 0

Note that directories can only be deleted if all ldp:contains resources were previously deleted. Also note that the Link header has to specify the type of the container and that according to RFC 5988 this has to be done with double quotes, so that the full curl command needs to be sent as single quote.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.