Clean up - data process api verification

[stlim0730]

We have lots of unnecessary -- also, not recommended to be distributed -- files in the repository. Actually, it's my fault that I didn't first share repo maintenance guideline with you, due to my trip and busy days at the beginning of this summer.

Also, issue #37 can't begin until this issue is resolved.

@sohyeonlee, @kbourque, @nutelIa, and @calo1, I may ask questions through this issue thread to learn what file is needed for what. I'm assigning myself for this issue.

[stlim0730]

Repository has been cleaned up. This version is running on Heroku. Please pay enough attention to all the features are working. This change is reversible if needed. Let me know if anything went wrong.

[calo1]

@stlim0730 I'll be testing throughout the day

[stlim0730]

@calo1 and @sohyeonlee, I think we should test api process. It's the only part we didn't see together yesterday.

[calo1]

@stlim0730 @sohyeonlee : The test should go like this.

1. Delete extraneous data_point / data_polygon tables on Carto

2. Use farmview admin to add data

3. check to see if data arrives

4. Then force sync

5. add data through Ona.io

6. see if data appears on Carto -> farmview

---

Then: Save the CSS and Html for data_point and data_polygon

Rename or delete data_point and data_polygon

Run Irene's tool through the admin again

end

[calo1]

@stlim0730 @sohyeonlee step 2 : getting the following error upon add new form data

[stlim0730]

So, the error doesn't seem to look for missing stuff that I accidentally removed, which is good.
As @sohyeonlee said, it sounds like a missing environmental variable; since the runtime environment has changed into Heroku server. Looks fixable. I'm on it.

For lookup:
https://devcenter.heroku.com/articles/heroku-local
https://devcenter.heroku.com/articles/config-vars#add-ons-and-config-vars

[stlim0730]

@sohyeonlee, is all I need to set DROPBOX_AUTH_TOKEN, CARTODB_API_KEY, and ONA_AUTH_TOKEN that api_process.py accesses?

[sohyeonlee]

@stlim0730 That is correct!

[calo1]

1. Delete extraneous data_point / data_polygon tables on Carto

Done

2. Use farmview admin to add data
   success

3. check to see if data arrives
   success

4. Then force sync
   I applied a force sync .. but didnt see my new data arrive on carto.. perhaps carto on lets you sync every 15 minutes?

5. add data through Ona.io

6. see if data appears on Carto -> farmview

Then: Save the CSS and Html for data_point and data_polygon

Rename or delete data_point and data_polygon

Run Irene's tool through the admin again

end

[sohyeonlee]

@calo1 yes, force sync wouldn't work unless you have at least 15mins (16?) gap in between syncs!

[calo1]

Great .. resuming testing

[stlim0730]

What does

Then: Save the CSS and Html for data_point and data_polygon

mean?

[calo1]

when I temporarily sever the current data_point anddata_polygon table Carto might forget how it is styled ... so when I replace the tables with the synced one, we want the exact same styling

[stlim0730]

Don't we want to test if it actually loses it or not, at some point?

[calo1]

I can do that.

[calo1]

Ok, force syncing is not bringing in the new data added on ONA. It might be a problem on my end with data entry ... ONA is showing the new record .. Carto is not. There is this 15 minute delay which slows down the testing ... I have a meeting for an hour and will resume trouble shooting then.

In the meantime .. what does the "current" box mean on the admin panel?

[stlim0730]

You may enter multiple entries of Form data (right?). The current field indicates the entry has been currently applied.

[stlim0730]

@sohyeonlee, @calo1 reported the data flow is not successful for now. Any suggestions?

[calo1]

I have an idea.. I might not understand how the tool works ... Where is the dropbox folder?

[stlim0730]

https://www.dropbox.com/
You have the login credentials in your email.

farmview folder looks what @sohyeonlee's code created.
application_resources is a temporary folder to test audio playback. Might be gone soon.

[calo1]

OK, Update:

1. In order to get new data into carto from Ona, the "add data" command must be run. Then , I can run a force sync on the old form table and then the original carto table will successfully show the new data. yay!

however, the tool will be then adding new data to carto every time want to update the application .. not ideal.

2. there is a naming problem: api_process should make geojsonfiles called data_point and data_polygon .. right now it is new_data_point.

[stlim0730]

Thank you for the report, @calo1. Sounds like a fixable bug. It seems that you ran the script on your command line. Did you? When do you not want to add new data if exist in updating the application (what does the update of the application mean, though)? -- sorry for so many questions.

@sohyeonlee, could you confirm that the first report about add data above could be the possible behavior of the app? Also, can you parameterize the option of add data only when new data entries exist? I thought force sync runs on the admin page rather than the command line. Can we bring the optional add data command into the web UI?

[calo1]

I did all of this through the admin, not the command line.

Update of the application means .. there are new submissions on ONA but not on carto and not on farmview. Farmview needs those submissions to be updated. I wouldn't want to immediately add data because some data might be junk or sensitive .. It allows private editing via Ona before submission.

[calo1]

The api_process should import to carto selectively ... let me put it this way:
The dropbox files need to be replaced and the
carto files need to be force synced

[stlim0730]

@calo1,

So, did you mean that the process is 1) you click add form data 2) check the data added, and 3) make force sync in the drop down? It sounds reasonable. When do you want to make force sync without adding new form data?

@sohyeonlee, I see multiple current entries. Is this expected?

[calo1]

1. you click add form data 2) check the data added, and 3) make force sync in the drop down?

Click add form data, go to dropbox and inspect geojson ... new data is added. check carto, two synced tables are now present the initial upload with say 9 rows and and a new entry with 10 rows.

make force sync on the drop down .... but on the original form data entry (48).

Both carto layers now have 10 rows and are up to date ...

But synced tables cant be renamed .. so they must be made with the desired name

[calo1]

update: sync tables can be renamed with SQL within carto, but then when you force sync that table, it becomes orphaned.

For example I changed the sync table new_data_point_1 to "taek_rules"

then I force synced through farmview dropdown ... taek rules still exists, but the sync function is broken

[stlim0730]

I think @calo1 and @sohyeonlee should meet tomorrow unless we can deal with this issue by this evening.. Does this sound okay?

[calo1]

Sounds good!

I think in reality we are really close, but we need a reliable workflow to get new data into farmview.

[sohyeonlee]

Hello, I am still outside and won't have time until tonight to take a look. So
the issue right now is that when running force_sync, new FormData is created?
(It was a lot to catch up correct me if i am wrong). I didn't rename
new_data_point yet since I didn't want to mess up with the live page. And
“current” field, honestly I didn't get what this was supposed to do so I didn't
really do anything with this but looks like what Taek wanted is to show the
latest data(?)

[calo1]

The issue is, running force sync doesn't add new data unless you first add a new table to carto ( vith add formdata). There is an unneeded redundancy.

All that really needs to happen is to have api_process make the table names data_point and data_polygon. That would solve the problem temporarily. ... then the redundancy stuff could be addressed later.

I think we can do this tonight with a backup thursday meeting

[sohyeonlee]

Hmmm I am still not sure if I get exactly what the problem is. New data table
should first be created by adding formdata, and force sync will be run on that
data set with the specific import_id (saved to formdata model when creating data
table).
When ona survey gets updated and force_sync is called, api_process should
reprocess the survey data with geoprocess.py and update the files in data set
under same names.. is it not the way it works right now?

[stlim0730]

It's @calo1's call, but I think it's good to address the redundancy issue. I see @sohyeonlee's concern in naming the tables, so It seems that what you want to do is 1) backup all the dataset on Carto and 2) test the process with the tables renamed. Do you think this will solve the redundancy issue?

Forget about current field for now.

[calo1]

This would not solve the redundancy issue .. but it would get the tool functional.

[calo1]

I don't think so ...add form is adding a new table each time ..because of carto naming conventions ..new data isn't passed to the old table. More in the morning!

[stlim0730]

Merged and closed.