New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moving the dji_pilot_groundStation_db_DJIWPCollectionItem database to user storage ? #95

Closed
Matioupi opened this Issue Nov 7, 2017 · 28 comments

Comments

Projects
None yet
5 participants
@Matioupi
Collaborator

Matioupi commented Nov 7, 2017

Hello,

I'm trying to track down where the app stores the mission files.
I tryed several methods : backup full phone, create and fly a mission, rebackup and check changed files...
From this first approach, it seems that the database is not in a file accessible to user from non rooted android. Is this possible that an app can read/write some files that stays hidden to non root user ?
While going thrhought the code, I noticed in

decompile_out/smali_classes4/dji/pilot/dji_groundstation/controller/DataMgr/f.smali

a function where there is the string:

const-string v0, "SELECT * FROM dji_pilot_groundStation_db_DJIWPCollectionItem"

Do you have a clue / can you help tracking down where this db is ?
Do you think it is possible to patch in order to move this database to "regular user storage" ?

Regards

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 7, 2017

There is another occurence of this db in decompile_out/smali_classes5/dji/pilot/publics/objects/DJIApplicationLike$1.smali

const-string v0, "ALTER TABLE dji_pilot_groundStation_db_DJIWPCollectionItem ADD COLUMN autoAddFlag INTEGER default 0;"

but those 2 are the only one I could find in decompiled files

@Bin4ry

This comment has been minimized.

Owner

Bin4ry commented Nov 7, 2017

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 7, 2017

thanks for that confirmation, I need to root my device now :-)

Do you know if this database is encrypted and if it's possible to add missions from external tools (once rooted) ?

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 7, 2017

Or would it be possible to patch the app another way like:

when starting or pressing a button, it parse a file in normal storage area and merge it within this hidden database ?

This would not requiere root and still provide the capability to design offline missions...

@Bin4ry

This comment has been minimized.

Owner

Bin4ry commented Nov 7, 2017

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 7, 2017

Unfortunately, programming something like this is beyond my skills (within Android Universe and my knowledge of DJI GO4 architecture...)...

But if this can give ideas to one of the brilliant modders around, that would be a great new feature of the modded App vs. regular App !

I'll still check if the database is encrypted or not as soon as possible.
Thank you for your answers !

@Bin4ry

This comment has been minimized.

Owner

Bin4ry commented Nov 7, 2017

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 9, 2017

I can confirm that the data is in the /data/data/dji.go.v4/databeses/dji.db
Rooted device allows to access it.

I have not been able to go out and set up a mission for testing but my first guess is that data is unencrypted and points coordinates/properties stored as a json string (I lost my previous missions while rooting my device.)

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 10, 2017

More details : points are stored in the

dji_pilot_dji_groundstation_controller_DataMgr_DJIWPCollectionItem database within /data/data/dji.go.v4/dji.db file

each record has the following fields

Id : integer start to 1
distance : flight plan length stored with 3 digits after dot e.g. : 175.561
pointsJsonStr : the actual flight plan : example :
{"points":[{"craftYaw":91,"gimbalPitch":0,"gimbalYaw":0,"height":12.199999809265137,"lat":45.79081293135456,"lng":2.8854028055682677},{"craftYaw":92,"gimbalPitch":0,"gimbalYaw":0,"height":14.40000057220459,"lat":45.79079626314489,"lng":2.8860285703542408},{"craftYaw":-128,"gimbalPitch":0,"gimbalYaw":0,"height":25.0,"lat":45.79050350040998,"lng":2.8860265170747117},{"craftYaw":-45,"gimbalPitch":0,"gimbalYaw":0,"height":25.0,"lat":45.790401616042686,"lng":2.8857248074387702},{"craftYaw":-82,"gimbalPitch":0,"gimbalYaw":0,"height":20.399999618530273,"lat":45.790523865365834,"lng":2.8855512560565972},{"craftYaw":-46,"gimbalPitch":0,"gimbalYaw":0,"height":20.399999618530273,"lat":45.79060532031695,"lng":2.8853148006888967},{"craftYaw":98,"gimbalPitch":0,"gimbalYaw":0,"height":20.399999618530273,"lat":45.790834350748256,"lng":2.8854142315025153}]}

location is empty on the flight plan I created from DJI GO 4

autoAddFlag is 1

createdDate is the Unix timestamp of the mission creation in milliseconds (can check and convert with this : https://www.epochconverter.com/ )

Now next step is to create a flight plan offline and try to fly it !

database is the

@SteveRan

This comment has been minimized.

SteveRan commented Nov 10, 2017

Outstanding!

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 17, 2017

A very small update to confirm that creating a new record in the database without DJI GO allows to use it from DJI GO...
After long time of bad weather or work travel, I just had a chance to test.

  1. I designed a fligh path from Google Earth and saved it to kml.
  2. I opened the kml file with notepad++ and extracted the data
  3. pasted that in a spreadsheet
  4. completed other columns (height, craft yaw etc.)
  5. saved it as csv
  6. reopenned the csv with notepad++ and removed all spaces add missing brackets so that it matches the pointJsonStr format
  7. Transfered that to the rooted android device and put it in clipboard
  8. Openned /data/data/dji.go.v4/dji.db with SQLite Editor app (version 2.2.1) which handles root access
  9. go to dji_pilot_dji_groundstation_controller_DataMgr_DJIWPCollectionItem subDatabase
  10. Create a new record (button +)
  11. Fill in "basic field" and paste the pointsJsonStr from previoulsy created data
  12. save
  13. go to the field and open DJI GO 4... After taking off and switching to waypoint mode, you'll be able to load the mission from the saved mission list

After coming back from field, it seems that the database can only hold 5 missions and the missions flown from the created ones are automatically added as new missions. Maybe the autoAddFlag can controll this behaviour. It is also possible it behaves as a kind of index as I have the value 2 or 3 for some missions I played twice...

The original mission I added with id 2 also now have id 6...

Most of the flight plan creation could be automatized to go straight from a kmz file...

log files of the 2 missions :

http://www.phantomhelp.com/logviewer/E9XKA3559LQA9SPWUIEO/

http://www.phantomhelp.com/logviewer/Z8UDW12ZUJO3VQNQGGBI/

@Pathogenius

This comment has been minimized.

Collaborator

Pathogenius commented Nov 17, 2017

Very impressive work!

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 18, 2017

I'm now wondering if it's possible to move the dji.db database to public space... that shoud be quite an easy patch as I'm pretty sure it only requiered changing the path of dji.db file in the open call.

That would allow this offline planning feature on non rooted devices.

I'll try to that next week

@Bin4ry

This comment has been minimized.

Owner

Bin4ry commented Nov 18, 2017

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 18, 2017

the string dji.db is found at 2 locations in .smali files

smali_classes2/com/dji/frame/c/c.smali
smali_classes6/dji/thirdparty/afinal/b$a.smali

the "opendatabase(" string can be found in 8 files, but never with a hardcoded string

smali_classes3\com\here\sdk\analytics\internal\h$1.smali
smali_classes3\com\here\sdk\analytics\internal\h.smali
smali_classes3\com\here\sdk\analytics\internal\SQLiteDatabaseConnect.smali
smali_classes6\dji\thirdparty\afinal\b.smali
smali_classes6\net\sqlcipher\database\SQLiteDatabase$2.smali
smali_classes6\net\sqlcipher\database\SQLiteDatabase$3.smali
smali_classes6\net\sqlcipher\database\SQLiteDatabase.smali
smali_classes6\net\sqlcipher\database\SQLiteOpenHelper.smali

the "openOrCreateDataBase(" can be found in 6 files
smali\dji\midware\data\forbid\util\FlyforbidUtils.smali
smali_classes2\com\amap\api\col\fi$a.smali
smali_classes2\com\amap\api\col\hk.smali
smali_classes3\com\mob\tools\utils\SQLiteHelper$SingleTableDB.smali
smali_classes6\net\sqlcipher\database\SQLiteDatabase.smali
smali_classes6\net\sqlcipher\database\SQLiteOpenHelper.smali

I've not been able to find "path strings" such as "/data/data"

If someone can help me tracking down from thoses clues, he/she is welcome !

If I want to try a patch changing path directly from the "dji.db" string (which according to
https://stackoverflow.com/questions/4452538/location-of-sqlite-database-on-the-device/21832714#21832714 has a chance to work)
how should I re-encrypt the new string for producing the patches ?

@Bin4ry

This comment has been minimized.

Owner

Bin4ry commented Nov 18, 2017

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 18, 2017

seems to be working !

for 4.1.9-2025608 version:

--- orig/smali_classes2/com/dji/frame/c/c.smali	2017-11-18 14:48:29.569826000 +0100
+++ mod/smali_classes2/com/dji/frame/c/c.smali	2017-11-18 14:58:01.170307104 +0100
@@ -382,11 +382,7 @@
 
     const/4 v1, 0x0
 
-    const-string v2, "PUAgbAM8"
-
-    invoke-static {v2}, Lcom/dji/k/a/a/a;->a(Ljava/lang/String;)Ljava/lang/String;
-
-    move-result-object v2
+    const-string v2, "/mnt/sdcard/dji_test_matioupi.db"
 
     sget-boolean v3, Lcom/dji/frame/c/c;->f:Z
 
--- orig/smali_classes6/dji/thirdparty/afinal/b$a.smali	2017-11-18 14:48:36.953826000 +0100
+++ mod/smali_classes6/dji/thirdparty/afinal/b$a.smali	2017-11-18 15:01:32.099714405 +0100
@@ -46,11 +46,7 @@
     iput-object v0, p0, Ldji/thirdparty/afinal/b$a;->a:Landroid/content/Context;
 
     .line 904
-    const-string v0, "PUAgbAM8"
-
-    invoke-static {v0}, Lcom/dji/k/a/a/a;->a(Ljava/lang/String;)Ljava/lang/String;
-
-    move-result-object v0
+    const-string v0, "/mnt/sdcard/dji_test_matioupi.db"
 
     iput-object v0, p0, Ldji/thirdparty/afinal/b$a;->b:Ljava/lang/String;
 


I have not flown it yet, but at least the file is properly created at user space storage root (on a non rooted device)

@Bin4ry

This comment has been minimized.

Owner

Bin4ry commented Nov 18, 2017

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 18, 2017

I just did a test flight and everything seems to work as it should with the databased moved !
I recorded and flew a short mission, the point fall in the new moved database !

As for doing a git pull i'll have to check how to do that... and prepare the phone version patch too.
Which path and moved database name do you think would be best ?

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 19, 2017

I wrote a very short (PC) command line utility to convert from
Ardupilot Mission planner mission file format to DJI JSON

http://ardupilot.org/planner/docs/common-install-mission-planner.html

I saw there are some Android versions of this mission planner available on the playstore.

Having a simple apk that would let you convert and load from a .waypoint (eitehr generated on PC or from one of the android versions) and then load it to the moved dji.db should be possible... However, I really have no skill in Android Development...

ArduMissionToDJIJsonSRC.zip
ArduMissionToDJIJson.zip

@sebip

This comment has been minimized.

sebip commented Nov 19, 2017

ArduMissionToDJIJson crash on win10 64

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 19, 2017

The exe I attached is a 64 bit version and I'm running it on W10 without issues.
Maybe this can help:

  • it's a command line utility (you need to call it from a console window)
  • you need to have MS VC redist 2015 64 bits installed on the PC : https://www.microsoft.com/fr-fr/download/details.aspx?id=48145
  • the command takes 2 arguments : path to the .waypoints input file (arg1) and patg to the json output file (arg2)
    example : ArduMissionToDJIJson L11.waypoints L11.json
@sebip

This comment has been minimized.

sebip commented Nov 19, 2017

with arguments it is working . thank you

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 19, 2017

I just slighlty enhanced to write directly a sql command that can then be executed with some Android Apps (e.g. SqlitePrime). This avoid cut and paste and makes things slightly easier...

It takes a third argument : the integer value of autoAddFlag (I did not figured out yet what it stands for)

the createdDate is filled from local PC clock at time of conversion from .waypoints file

ArduMissionToDJISQL.zip

this is not taking care if there are more than 5 entries in the dji_pilot_dji_groundstation_controller_DataMgr_DJIWPCollectionItem table, and I don't know yet if DJI GO 4 app is working when the table contains more than 5 entries in this table.

@sebip

This comment has been minimized.

sebip commented Nov 21, 2017

You can put here the link to your tutorial from dropbox ...

@Bin4ry

This comment has been minimized.

Owner

Bin4ry commented Nov 21, 2017

@Matioupi

This comment has been minimized.

Collaborator

Matioupi commented Nov 21, 2017

The docx Office version in case someone wants to improve the document:

https://www.dropbox.com/s/k0awh8apgwmkwaf/Tutorial%20DJI%20Waypoints%20with%20Spark.docx?dl=0

@Bin4ry Bin4ry closed this Nov 25, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment