Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Laser Reader Freezing #15

Open
MakerLabs-Van opened this issue Feb 15, 2019 · 24 comments
Open

Laser Reader Freezing #15

MakerLabs-Van opened this issue Feb 15, 2019 · 24 comments

Comments

@MakerLabs-Van
Copy link

The reader on the laser cutter would freeze on the "searching" function.

It would respond when the tag was removed or placed on the reader, but would never complete the search.

Tried to restart 3 times unsuccessful, then the reader rebooted on it's own. After 1 unsuccessful search it was able find the user and started the safety systems.
@paulreimer
Copy link
Collaborator

Do you know the time range when this occurred? And ideally the Maker Name or Tag ID used?

As well, can you describe what happened during the "unsuccessful search" behaviour?

@MakerLabs-Van
Copy link
Author

MakerLabs-Van commented Feb 15, 2019 via email

@paulreimer
Copy link
Collaborator

Also, during "Tried to restart 3 times unsuccessful, then the reader rebooted on it's own", what were the messages displayed on the screen at that time?

@MakerLabs-Van
Copy link
Author

The reader was restarted:

ACM Logged In appeared on the screen, then when presented with the tag would search.

when it restarted on it's own:

  • it displayed the ACM v.1.X.X (I don't remember the version number)

  • It flashed and displayed a illegible mash that looked like the user name and something else,

  • Then it displayed "Logged In"

  • Then it began to search for the user.

  • Note during that reboot the safety systems didn't switch on, so power didn't seem to have been cut out.

@paulreimer
Copy link
Collaborator

OK. Just to confirm the time range, do you mean 12:50 ~ 1:03?

And, was the reader manually restarted the 3 times (using the reset button?)

So, the behaviour was:

  1. Could not successfully get info for tag (Tag ___ Search... -> 15s timeout)
  • Including repeated scans (and to confirm, each time it would beep after reading the tag and start a new 15s progress bar? Did it also beep each time the tag was removed?)
  1. Pressed reset button (or cycled USB power) 3 times, same behaviour as (1)
  2. Rebooted on its own (potentially unrelated to the manual resets)
  3. Failed initial tag search
  4. System normal on second tag search

I can say that (4) may be expected after a reboot (only the first tag read after reset may not trigger operation).
And (1) + (2) sound like a network issue (which would eventually lead to (3), automatic reboot). Do you know if other wifi devices were working at that time?

@paulreimer
Copy link
Collaborator

It looks like this made it to the spreadsheet -- does "Activity, De-duped", starting at row 7308, describe the time range? (in that case: not a networking issue but something else.)

@MakerLabs-Van
Copy link
Author

Yes, that is the correct time range. I don't know if all the operations made it to the time sheet, because we tried scanning the tags many times,

  • the first two reboots were unplugging the USB power, the third using the reset button.
  • The repeated scans would restart the progress bar

@MakerLabs-Van
Copy link
Author

This is consistently happening with multiple members and FOBS, both guest and assigned. It's also not sending the signal for the safety systems to turn off when the FOB is removed.

@paulreimer
Copy link
Collaborator

OK, I have modified the logging from the attached Raspberry Pi's to be more durable. I don't have the logs for the time of the original issue, but I may have the logs for the more recent occurrences.

If you know the time ranges when this occurs (as well as times when it is reset manually), that would be most helpful. I'll keep an watching the (new) logs to see if it happens again.

@paulreimer
Copy link
Collaborator

Just curious if this issue has re-occurred at all? I have all the logs since the previous message, and I believe I can leave it enable for the near future (I think the amount of logs is within the free quota).

@MakerLabs-Van
Copy link
Author

MakerLabs-Van commented Feb 22, 2019 via email

@paulreimer
Copy link
Collaborator

paulreimer commented Feb 22, 2019 •
edited
Loading

Hey this is really interesting. Glad I have some logs to look at, but at the time you mention, the Raspberry Pi that I have connected to the device had just finished rebooting (this is not expected to happen except e.g. in a power outage).

Meanwhile, throughout the logs there is a recurring message about the Raspberry Pi running into a low-voltage condition.

Both these things make me suspicious of the USB brick that is powering the Pi (and by extension the TARS reader unit, which is being powered from the Pi, for remote logging purposes).

If there is a good quality USB power brick for the Pi, that may fix some of these issues. (The "Low Voltage" condition warnings should stop, as a confirmation of that test). Could you send a picture of what is powering that Pi? I believe at least 5V@2A is required, and 3A would be even better.

@MakerLabs-Van
Copy link
Author

MakerLabs-Van commented Feb 22, 2019 via email

@MakerLabs-Van
Copy link
Author

This is still happening on a regular basis, it happened today with tag ID #13952. I don't know the sequence, but it normally happens and we restart the reader 1 to 3 times, and then it has a 15-30 seconds to reboot and then search before registering the tag and allowing the laser cutting.

@MakerLabs-Van
Copy link
Author

After it having trouble finding my tag and freezing today at approximately 11:55am, we have taken the laser fob system offline and are running the bypass to use the laser cutter.

@paulreimer
Copy link
Collaborator

paulreimer commented Apr 8, 2019 •
edited
Loading

Noted.

I have some ideas on how to approach these reliability issues.

The simplest approach would involve modifying the firmware to switch from HTTP to MQTT (still re-using AWS lambda functions -> Google services as before). It is a more efficient use of the network connection, and the device logic becomes simpler. I can look into estimating the time it would take to perform these changes.

An alternate approach would be to modify the reader device to integrate more closely with the Windows PC that controls the laser cutter, and for the reader device to forward scanned tag IDs to the PC to use its much faster wifi connection (and perhaps display a popup on the PC screen with the User info when someone logs in, and/or completes a job). This is more complicated than the above suggestion (HTTP -> MQTT), but does offer more features by leveraging the PC.

Both of these approaches have the potential to resolve this issue (#15), as well as #13 and #16 .

@MakerLabs-Van
Copy link
Author

MakerLabs-Van commented Apr 9, 2019 via email

@MakerLabs-Van
Copy link
Author

MakerLabs-Van commented Apr 17, 2019 via email

@paulreimer
Copy link
Collaborator

paulreimer commented Apr 22, 2019 •
edited
Loading

Hi Marc, all,

Given the amount of risk/uncertainty in estimating research work for this solution, it is not possible to make an estimate until approximately the 80% completion mark. To that end, I can report that the switch to MQTT should take a total time of approximately 20~25 hours including install time (but not including new issues discovered as a result of installation/testing), of which 16 hours have already been completed.

At this time the preliminary results with MQTT show the reliability should improve (overall there are fewer network connections used and it does not need to wait for a response between each scan). Performance is also improved approx. 2x for the time-tracking use-case (e.g. laser cutter units), and is compatible with the previous HTTP-based system (e.g. the Front Desk can continue to use the existing HTTP approach if desired)

Please let me know if you would like me to continue with the implementation, or investigate other possible issues, and/or if you would like to explore the other option (i.e. using the Windows laptops for internet/WiFi traffic)

@MakerLabs-Van
Copy link
Author

MakerLabs-Van commented Apr 23, 2019 via email

@paulreimer
Copy link
Collaborator

Hi Marc,

Yes, there is still under-voltage condition regularly on the Pi. You could try a beefier power supply to the Pi, and I can let you know if the power quality is improved (e.g. the error goes away).

The Pi is only for out-of-band monitoring, and could be removed (so the reader units could be powered by the DC barrel jack -- no Pi or USB cable should be used in that case). You could also try that (now, before, or after the 3A supply tests) to see if this is issue goes away permanently.

FYI I have not yet found the Pi to be useful in having useful logs near the time when something was reported, but without the Pi I would be unable to check into things. But you could take the Pi out -- try it, and if the problem ever comes up again (then it wasn't necessarily the under-voltage issue as the root cause), then put the Pi back in.

@MakerLabs-Van
Copy link
Author

MakerLabs-Van commented Apr 23, 2019 via email

@paulreimer
Copy link
Collaborator

OK, if you let me know when the power supply has been swapped out, I can let you know if the under-voltage warning is still being logged.

@MakerLabs-Van
Copy link
Author

MakerLabs-Van commented Apr 24, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@paulreimer @MakerLabs-Van