Importing JSON databases through the REST API

[whatyouhide]

No description provided.

[whatyouhide]

The documentation around this is extremely scarce (it's here) and slightly confusing (e.g., it says to POST to http://<server>:[<port>]/import/<database> but then in the example it POSTs to http://localhost:2480/import/).

Let's assume I connected to the MarcoPoloTest database, like this:

curl --user root:root -X GET 'http://localhost:2480/connect/MarcoPoloTest' \
  -H 'Accept-Encoding: gzip,deflate'

and that I wrote down the OSESSIONID cookie set by the response.

Let's assume we have a file called db that is the result of exporting the MarcoPoloTest database, like this:

curl --user root:root -X GET 'http://localhost:2480/export/MarcoPoloTest' \
  -H 'Accept-Encoding: gzip,deflate' \
  -H 'Content-Type: application/json' \
  --cookie 'OSESSIONID=...' \
  > db.gzip

Then, let's assume I've unzipped db.gzip to db and I'm connected to the ImportDest database by doing:

curl -v --user root:root -X GET 'http://localhost:2480/connect/ImportDest' \
  -H 'Accept-Encoding: gzip,deflate'

HTTP/1.1 204 OK
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Date: Fri Mar 11 13:30:53 CET 2016
Content-Type: text/plain; charset=utf-8
Server: OrientDB Server v.2.2.0-beta (build develop@r4cbc24a66f17f023a2e59da300a29b91fdb5d1fa; 2016-02-22 17:46:44+0000)
Connection: Keep-Alive
Set-Cookie: OSESSIONID=OS1457699453268-4591488868473251284; Path=/; HttpOnly
Content-Length: 0

If I make a POST request to http://localhost:2480/import/ImportDest with Content-Type: application/json and the JSON-encoded database in the body (which is what I expected to work), it complains that the request is not multipart/form-data (still not sure why it should be, but these lines in the OrientDB source confirm that the check is made there):

curl -v --user root:root -X POST 'http://localhost:2480/import/ImportDest' \
  -H 'Accept-Encoding: gzip,deflate' \
  -H 'Content-Type: application/json' \
  --cookie 'OSESSIONID=OS1457699453268-4591488868473251284' \
  -d @db

HTTP/1.1 405 Request is not multipart/form-data
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Date: Fri Mar 11 13:32:53 CET 2016
Content-Type: text/plain; charset=utf-8
Server: OrientDB Server v.2.2.0-beta (build develop@r4cbc24a66f17f023a2e59da300a29b91fdb5d1fa; 2016-02-22 17:46:44+0000)
Connection: Keep-Alive
Set-Cookie: OSESSIONID=OS1457699453268-4591488868473251284; Path=/; HttpOnly
Content-Length: 34

Request is not multipart/form-data

If I do what the error message suggests and change the content-type to multipart/form-data by uploading a file, I still get an error:

curl -v -X POST 'http://localhost:2480/import/ImportDest' \
  -H 'Accept-Encoding: gzip,deflate' \
  --cookie 'OSESSIONID=OS1457703457177-2218910698106246248' \
  -F file=@db

HTTP/1.1 405 Wrong request: Expected CR/LF
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Date: Fri Mar 11 14:40:14 CET 2016
Content-Type: text/plain; charset=utf-8
Server: OrientDB Server v.2.2.0-beta (build develop@r4cbc24a66f17f023a2e59da300a29b91fdb5d1fa; 2016-02-22 17:46:44+0000)
Connection: Keep-Alive
Set-Cookie: OSESSIONID=OS1457703457177-2218910698106246248; Path=/; HttpOnly
Content-Length: 29
* HTTP error before end of send, stop sending

Wrong request: Expected CR/LF

This error should come from here.

Apparently, the import does something on the server. This is the server log:

2016-03-11 16:44:25:647 INFO  {db=ImportDest} Started import of database 'plocal:/Users/whatyouhide/.odbvm/installations/orientdb-community-2.2.0-beta/databases/ImportDest' from
streaming.json... [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:650 INFO  {db=ImportDest} Non merge mode (-merge=false): removing all default non security classes [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:675 INFO  {db=ImportDest} - Class E was removed. [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:690 INFO  {db=ImportDest} - Class V was removed. [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:692 INFO  {db=ImportDest} - Class OTriggered was removed. [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:694 INFO  {db=ImportDest} - Class ORestricted was removed. [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:698 INFO  {db=ImportDest} - Class OSequence was removed. [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:702 INFO  {db=ImportDest} - Class OSchedule was removed. [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:714 INFO  {db=ImportDest} - Class OFunction was removed. [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:716 INFO  {db=ImportDest} Removed 7 classes. [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:719 INFO  {db=ImportDest} Rebuild of stale indexes... [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:719 INFO  {db=ImportDest} Start rebuild index ouser.name [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:726 INFO  {db=ImportDest} - Rebuilding index ImportDest.OUser.name (estimated 3 items)... [OIndexRebuildOutputListener]
2016-03-11 16:44:25:730 INFO  {db=ImportDest} --> OK, indexed 3 items in 4 ms [OIndexRebuildOutputListener]
2016-03-11 16:44:25:730 INFO  {db=ImportDest} Rebuild  of index ouser.name is completed. [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:730 INFO  {db=ImportDest} Start rebuild index orole.name [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:733 INFO  {db=ImportDest} - Rebuilding index ImportDest.ORole.name (estimated 3 items)... [OIndexRebuildOutputListener]
2016-03-11 16:44:25:734 INFO  {db=ImportDest} --> OK, indexed 3 items in 2 ms [OIndexRebuildOutputListener]
2016-03-11 16:44:25:734 INFO  {db=ImportDest} Rebuild  of index orole.name is completed. [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:734 INFO  {db=ImportDest} Stale indexes were rebuilt... [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:739 INFO  {db=ImportDest} Deleting RID Mapping table... [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:740 INFO  {db=ImportDest} OK
 [OServerCommandPostImportDatabase]
2016-03-11 16:44:25:740 INFO  {db=ImportDest}
Database import completed in 93 ms [OServerCommandPostImportDatabase]

but the actual import doesn't seem to work as I can't see the classes from MarcoPoloTest in the ImportDest database.

This is taking me a quite long time, so it would be super-nice if someone from @MyMedsAndMe/voyagers could help in finding a curl command that works here :). I tried asking @tglman but he's scarcely available these days, so I can bother him just every once in a while 😄. Let's go crazy and ping @lvca as well since the blame for the lines that raise the Wrong request: Expected CR/LF error points to him. Thanks everyone in advance!

[yelps]

Hey Andrea,
I suppose it's related to the same problem I had before. The reason behind the weird error message is that http import doesn't support the "- merge" option and ends up in a indiscriminate state.

There is a request open for this issue:
http://www.prjhub.com/#/issues/5923

It suppose to be fixed in OrientDB 2.2 Beta which is available here:

http://orientdb.com/orientdb-labs/

[whatyouhide]

Hey @yelps, I assumed this wasn't related to the -merge option as I'm importing a dumped database to a freshly created database, so there should be no need for any kind of merging. Also, I am indeed running on 2.2-beta (this can be seen in the Server header in the responses above, which is OrientDB Server v.2.2.0-beta (build develop@r4cbc24a66f17f023a2e59da300a29b91fdb5d1fa; 2016-02-22 17:46:44+0000)).

[yelps]

Oh, I see. Then it's probably best to ask the support team of Orient Technologies.

[simonewebdesign]

I managed to reproduce the same error:

curl --user root:secret -X POST 'http://localhost:2480/import/demo_db' \
      -H 'Accept-Encoding: gzip,deflate' \
      -H 'Content-Type: multipart/form-data' \
      --cookie 'OSESSIONID=OS1457699453268-4591488868473251284; Path=/; HttpOnly' \
      --form "filename=@db.gzip"

Wrong request: Expected CR/LF⏎

I tried decompressing the gzip file, converting LF to CR LF while recompressing it, but still no luck.

Anyway I think this is something that should be addressed in the OrientDB repo, as I would expect the REST API to be able to accept application/json in the first place.

[whatyouhide]

@simonewebdesign absolutely, it makes much more sense to POST a JSON database as just JSON binary with Content-Type: application/json instead of having to go through the hustle of encoding it as multipart.

[whatyouhide]

This is on hold waiting for @tglman to let us know when a release including the changes described in orientechnologies/orientdb#5837 and implemented in orientechnologies/orientdb@253e3cf will be released in a new OrientDB 2.2 beta version.

[tglman]

OrientDB 2.2 beta2 has been released.
here the link: http://orientdb.com/download.php?email=unknown@unknown.com&file=orientdb-community-2.2.0-beta.tar.gz&os=multi

[whatyouhide]

This was hopefully implemented in e567aef. I don't fully trust the OrientDB side yet, but I think we're doing everything right on the client (MarcoPolo) side, so I'm gonna close this for now.