New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add field scanner operational log to data stream; add additional information? #128

Closed
dlebauer opened this Issue Jul 14, 2016 · 24 comments

Comments

Projects
None yet
6 participants
@dlebauer
Member

dlebauer commented Jul 14, 2016

Description

According to @smarshall-bmr the field scanner generates a log file. Can this be added to the data stream? It could contain information like start time, stop time, errors, etc.

There may be a list of additional information that could be added, but for starters it would be great if this log could be added to the data stream. Does it contain any sensitive information?

Planning

  1. Scheduling scans and field operations to overlap. Currently we have this calendar
    • This is a tool for planning, but doesn't need to be used to record what actually happened
  2. Sensor Campaign and Missions

Machine Logs

  1. Scripts used
  2. logs automatically generated by system
    • don't need to be neat, but are they collecting our information

Operator Logs

Can be created as one or more google forms

  1. gantry operations
  2. Agronomic management
  3. field data collection

Currently we have

  1. FieldBook spreadsheet
  2. Also some draft forms these need to be modified
  3. Github repository for scripts (currently private until LemnaTec agrees or censors content) ask David for access

The FieldBook is a catch-all that can be used to determine what the forms should collect.

Requests for feedback?

What needs to be done and who can do it?
What other information is available, could be recorded?

@smarshall-bmr

This comment has been minimized.

Show comment
Hide comment
@smarshall-bmr

smarshall-bmr Jul 14, 2016

Collaborator

The PLC log doesn't contain start or stop times. It's more of a safety record of enabling the laser and PS2 and any errors that have occurred. It would likely be most useful for determining down time.

Collaborator

smarshall-bmr commented Jul 14, 2016

The PLC log doesn't contain start or stop times. It's more of a safety record of enabling the laser and PS2 and any errors that have occurred. It would likely be most useful for determining down time.

@smarshall-bmr

This comment has been minimized.

Show comment
Hide comment
@smarshall-bmr

smarshall-bmr Jul 25, 2016

Collaborator

@TinoDornbusch would you happen to know where this log is stored on the PLC computer? I think it would be useful to pull a section of the log data so we can decide what we would like to do with it.

@dlebauer When we talk about adding this to the data stream, do we mean to store this with the field activity log? I have heard conflicting reports on if the gantry log and field log are going to be split or if the PLC log will be integrated with the gantry log. I would say both of those are good ideas, but I'm not sure what organizational scheme would be most useful to the folks downstream.

Collaborator

smarshall-bmr commented Jul 25, 2016

@TinoDornbusch would you happen to know where this log is stored on the PLC computer? I think it would be useful to pull a section of the log data so we can decide what we would like to do with it.

@dlebauer When we talk about adding this to the data stream, do we mean to store this with the field activity log? I have heard conflicting reports on if the gantry log and field log are going to be split or if the PLC log will be integrated with the gantry log. I would say both of those are good ideas, but I'm not sure what organizational scheme would be most useful to the folks downstream.

@dlebauer

This comment has been minimized.

Show comment
Hide comment
@dlebauer

dlebauer Jul 26, 2016

Member

@smarshall-bmr by 'add it to the data stream' I meant 'put it on the cache server and have it transferred alongside the rest of the data' ... at appropriate intervals (daily?) and without duplication.

Member

dlebauer commented Jul 26, 2016

@smarshall-bmr by 'add it to the data stream' I meant 'put it on the cache server and have it transferred alongside the rest of the data' ... at appropriate intervals (daily?) and without duplication.

@rachelshekar

This comment has been minimized.

Show comment
Hide comment
@rachelshekar

rachelshekar Sep 22, 2016

Contributor

@jdmaloney - are we receiving this data?

Contributor

rachelshekar commented Sep 22, 2016

@jdmaloney - are we receiving this data?

@jdmaloney

This comment has been minimized.

Show comment
Hide comment
@jdmaloney

jdmaloney Sep 22, 2016

Contributor

@rachelshekar I was unaware of this issue, I don't think we are as I haven't seen it explicitly in the directory structure that I can remember.

@smarshall-bmr If the data can make it to a folder on the cache server via FTP, I can make sure it gets moved to ROGER automatically, that last part is not hard to achieve at all. Just let me know where it's getting dropped in directory-wise.

Contributor

jdmaloney commented Sep 22, 2016

@rachelshekar I was unaware of this issue, I don't think we are as I haven't seen it explicitly in the directory structure that I can remember.

@smarshall-bmr If the data can make it to a folder on the cache server via FTP, I can make sure it gets moved to ROGER automatically, that last part is not hard to achieve at all. Just let me know where it's getting dropped in directory-wise.

@smarshall-bmr

This comment has been minimized.

Show comment
Hide comment
@smarshall-bmr

smarshall-bmr Sep 23, 2016

Collaborator

I'm having trouble finding where these logs are stored, any help @TinoDornbusch ?

Collaborator

smarshall-bmr commented Sep 23, 2016

I'm having trouble finding where these logs are stored, any help @TinoDornbusch ?

@TinoDornbusch

This comment has been minimized.

Show comment
Hide comment
@TinoDornbusch

TinoDornbusch Oct 5, 2016

Contributor

@smarshall-bmr Sorry I did not see that. The logs are saved on the control PC of the PLC. You can access it via remote desktop. The windows bar is hidden on the left. Mouse over opens the menu and you can start Windows Explorer. You can either Copy&Paste via remote desktop or use an external USB device to copy data.

Contributor

TinoDornbusch commented Oct 5, 2016

@smarshall-bmr Sorry I did not see that. The logs are saved on the control PC of the PLC. You can access it via remote desktop. The windows bar is hidden on the left. Mouse over opens the menu and you can start Windows Explorer. You can either Copy&Paste via remote desktop or use an external USB device to copy data.

@rachelshekar

This comment has been minimized.

Show comment
Hide comment
@rachelshekar

rachelshekar Jan 12, 2017

Contributor

@jdmaloney - are we getting this data now?

Contributor

rachelshekar commented Jan 12, 2017

@jdmaloney - are we getting this data now?

@jdmaloney

This comment has been minimized.

Show comment
Hide comment
@jdmaloney

jdmaloney Jan 12, 2017

Contributor

@rachelshekar No we are not

Contributor

jdmaloney commented Jan 12, 2017

@rachelshekar No we are not

@smarshall-bmr

This comment has been minimized.

Show comment
Hide comment
@smarshall-bmr

smarshall-bmr Jan 13, 2017

Collaborator

The active log .csv isn't able to be modified while the gantry is operating but I do have one "full" log that runs from 2/15/16 until 6/3/2016. I pulled a copy using a USB drive, any particular place I should put it on the cache server @jdmaloney ?

Collaborator

smarshall-bmr commented Jan 13, 2017

The active log .csv isn't able to be modified while the gantry is operating but I do have one "full" log that runs from 2/15/16 until 6/3/2016. I pulled a copy using a USB drive, any particular place I should put it on the cache server @jdmaloney ?

@smarshall-bmr

This comment has been minimized.

Show comment
Hide comment
Collaborator

smarshall-bmr commented Feb 2, 2017

@jdmaloney

This comment has been minimized.

Show comment
Hide comment
@jdmaloney

jdmaloney Feb 2, 2017

Contributor

@smarshall-bmr Sorry this issue slipped by me, we can get this one from Google Drive. Future logs can go in: /gantry_data/LemnaTec/gantry_log on the gantry-cache machine. Does that work for you?

Contributor

jdmaloney commented Feb 2, 2017

@smarshall-bmr Sorry this issue slipped by me, we can get this one from Google Drive. Future logs can go in: /gantry_data/LemnaTec/gantry_log on the gantry-cache machine. Does that work for you?

@smarshall-bmr

This comment has been minimized.

Show comment
Hide comment
@smarshall-bmr

smarshall-bmr Feb 2, 2017

Collaborator

@jdmaloney That should work well. I'm anticipating that the system will spit out a new log file soon given the first one covers about 4 months.

Collaborator

smarshall-bmr commented Feb 2, 2017

@jdmaloney That should work well. I'm anticipating that the system will spit out a new log file soon given the first one covers about 4 months.

@max-zilla

This comment has been minimized.

Show comment
Hide comment
@max-zilla

max-zilla Mar 8, 2017

Contributor

@smarshall-bmr are there new logs available? can you move them into the cache directory @jdmaloney mentioned if so (might be a good idea to make this automatic if possible)?

Contributor

max-zilla commented Mar 8, 2017

@smarshall-bmr are there new logs available? can you move them into the cache directory @jdmaloney mentioned if so (might be a good idea to make this automatic if possible)?

@smarshall-bmr

This comment has been minimized.

Show comment
Hide comment
@smarshall-bmr

smarshall-bmr Mar 9, 2017

Collaborator

@max-zilla the newest log file is still only 70% full. I've found that if I restart the HMI computer it takes a bit for the HMI to actually start up and in that time I can save a copy of the log file. If someone is interested in looking at the log data before it's automatically kicked out I can do this.

Collaborator

smarshall-bmr commented Mar 9, 2017

@max-zilla the newest log file is still only 70% full. I've found that if I restart the HMI computer it takes a bit for the HMI to actually start up and in that time I can save a copy of the log file. If someone is interested in looking at the log data before it's automatically kicked out I can do this.

@max-zilla

This comment has been minimized.

Show comment
Hide comment
@max-zilla

max-zilla Mar 30, 2017

Contributor

@smarshall-bmr any updates on this issue?

Contributor

max-zilla commented Mar 30, 2017

@smarshall-bmr any updates on this issue?

@jdmaloney

This comment has been minimized.

Show comment
Hide comment
@jdmaloney

jdmaloney Mar 30, 2017

Contributor

@max-zilla @smarshall-bmr As we just discussed in today's meeting, we'll just wait to receive this file until it fills up each time

Contributor

jdmaloney commented Mar 30, 2017

@max-zilla @smarshall-bmr As we just discussed in today's meeting, we'll just wait to receive this file until it fills up each time

@dlebauer

This comment has been minimized.

Show comment
Hide comment
@dlebauer

dlebauer Mar 31, 2017

Member

@jdmaloney Could you review the original task and propose an approach to putting all of the different sources of field log information into one location? The PLC logs are only one part, the two other crucial sources of information are the field book and field scanner log

@smarshall-bmr are you still using git commits to record changes to the scripts that define gantry operation?

Member

dlebauer commented Mar 31, 2017

@jdmaloney Could you review the original task and propose an approach to putting all of the different sources of field log information into one location? The PLC logs are only one part, the two other crucial sources of information are the field book and field scanner log

@smarshall-bmr are you still using git commits to record changes to the scripts that define gantry operation?

@smarshall-bmr

This comment has been minimized.

Show comment
Hide comment
@smarshall-bmr

smarshall-bmr Apr 5, 2017

Collaborator

@max-zilla I'm assuming that these files are size limited, in that case the one currently being used is about 80% full. The few errors that happen the longer it will take to fill up and the HMI is much better behaved now then it has been, so it might take a while.

@dlebauer I'll have to get help on the Git commands again at some point. I made the mistake of updating Git to a new version and it now wants me to update some other programs in order to function correctly. (We have updates turned off on the LemnaControl computer I'll talk to LemnaTec about how recent this install is, we might have to operate an outdated version of Git to get it to work.)

Collaborator

smarshall-bmr commented Apr 5, 2017

@max-zilla I'm assuming that these files are size limited, in that case the one currently being used is about 80% full. The few errors that happen the longer it will take to fill up and the HMI is much better behaved now then it has been, so it might take a while.

@dlebauer I'll have to get help on the Git commands again at some point. I made the mistake of updating Git to a new version and it now wants me to update some other programs in order to function correctly. (We have updates turned off on the LemnaControl computer I'll talk to LemnaTec about how recent this install is, we might have to operate an outdated version of Git to get it to work.)

@jdmaloney

This comment has been minimized.

Show comment
Hide comment
@jdmaloney

jdmaloney Apr 6, 2017

Contributor

@dlebauer I think there should be a distinction between the PLC logs and those other two inputs. Having @smarshall-bmr FTP the log over once it fills is a good way for us to get that information.

The other two inputs are more human driven and consistent in content types (columns), which to me indicates those should just get inserted into a database. I think our most logical direction would be to have one of the students create a simple web form for both the field book and the field scanner log, each submission of the form would insert an entry into a database that Max and I stand up and backup regularly. Trying to scrape .csv files out of Google Docs (not even sure the feasibility of that) is adding extra work in the middle, when the time comes to reference this data later, having a programmatically query-able data structure will be what we need anyway.

If there is a student who can work on this, I can work with them so they understand the objective and we get this stood up. I'm still contemplating where we'd run this service for maximum availability, but I should hopefully have that figured out here soon.

Contributor

jdmaloney commented Apr 6, 2017

@dlebauer I think there should be a distinction between the PLC logs and those other two inputs. Having @smarshall-bmr FTP the log over once it fills is a good way for us to get that information.

The other two inputs are more human driven and consistent in content types (columns), which to me indicates those should just get inserted into a database. I think our most logical direction would be to have one of the students create a simple web form for both the field book and the field scanner log, each submission of the form would insert an entry into a database that Max and I stand up and backup regularly. Trying to scrape .csv files out of Google Docs (not even sure the feasibility of that) is adding extra work in the middle, when the time comes to reference this data later, having a programmatically query-able data structure will be what we need anyway.

If there is a student who can work on this, I can work with them so they understand the objective and we get this stood up. I'm still contemplating where we'd run this service for maximum availability, but I should hopefully have that figured out here soon.

@dlebauer

This comment has been minimized.

Show comment
Hide comment
@dlebauer

dlebauer Apr 6, 2017

Member

@jdmaloney you can programmatically get the field scanner log as:

curl -o filed_scanner_log.csv 'https://docs.google.com/spreadsheets/d/1eQSeVMPfrWS9Li4XlJf3qs2F8txmddbwZhjOfMGAvt8/pub?gid=1394972273&single=true&output=csv' -L 
curl -o filed_book.csv 'https://docs.google.com/spreadsheets/d/1eQSeVMPfrWS9Li4XlJf3qs2F8txmddbwZhjOfMGAvt8/pub?gid=665425213&single=true&output=csv' -L

There are also libraries in R (googlesheets) and python (gspread) that provide more advanced functionality.

Member

dlebauer commented Apr 6, 2017

@jdmaloney you can programmatically get the field scanner log as:

curl -o filed_scanner_log.csv 'https://docs.google.com/spreadsheets/d/1eQSeVMPfrWS9Li4XlJf3qs2F8txmddbwZhjOfMGAvt8/pub?gid=1394972273&single=true&output=csv' -L 
curl -o filed_book.csv 'https://docs.google.com/spreadsheets/d/1eQSeVMPfrWS9Li4XlJf3qs2F8txmddbwZhjOfMGAvt8/pub?gid=665425213&single=true&output=csv' -L

There are also libraries in R (googlesheets) and python (gspread) that provide more advanced functionality.

@rachelshekar

This comment has been minimized.

Show comment
Hide comment
@rachelshekar

rachelshekar Apr 26, 2017

Contributor

@jdmaloney please update this issue

Contributor

rachelshekar commented Apr 26, 2017

@jdmaloney please update this issue

@max-zilla

This comment has been minimized.

Show comment
Hide comment
@max-zilla

max-zilla May 3, 2017

Contributor

@jdmaloney can we close this? is this just a cron thing we have running/need to run?

Contributor

max-zilla commented May 3, 2017

@jdmaloney can we close this? is this just a cron thing we have running/need to run?

@jdmaloney

This comment has been minimized.

Show comment
Hide comment
@jdmaloney

jdmaloney May 4, 2017

Contributor

@max-zilla We'll just be getting a manual push from @smarshall-bmr once the file fills (which isn't a set time). We're watching the destination on the cache server so we'll get it when we get it. I am closing the ticket.

Contributor

jdmaloney commented May 4, 2017

@max-zilla We'll just be getting a manual push from @smarshall-bmr once the file fills (which isn't a set time). We're watching the destination on the cache server so we'll get it when we get it. I am closing the ticket.

@jdmaloney jdmaloney closed this May 4, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment