quality problem with a series of AGTs?

[jerabaul29]

Hi,

I am building a large series of OpenMetBuoys (35) built using the AGT. It looks like we are facing some quality problems with the AGT unfortunately :( . This is a bit similar to what we had reported on sparkfunX/Artemis_Global_Tracker#29 .

More specifically:

• one of the AGTs could not be programmed over USB when I took it out of its protective bag. Upon visual inspection of the AGT, I quickly found the issue: this is on the chip between the USB-C connector and the artemis module: as you can see, there was a blob of solder that was making a short (picture from my smartphone, so not super good, but this is well visible):

luckily I was able to work with the finest tip of my soldering iron, and to clean this blob of solder. After this, the AGT works just fine and can be programmed. This is fine in the end, but for an less experienced end user this may look like a "bricked" AGT...

• I have several issues with 6 other AGTs. It seems that they are working only intermittently. I did not have as much time to debug these as I wanted, but I looked into 1 in particular. The issue is intermittent: if I take the AGT out of my lab (20 degrees C) and put it outside, it works normally for a while. Then after a few hours out in the cold (-5 degrees C), the GPS becomes unable to get a fix, so I suppose that the GPS signal does not reach the GPS chip or something like this (I can see the blue GPS LED turn on, and the GPS communication is working fine, but there is no GPS fix obtained even after 20 minutes). This seems to be quite "reproducible": I took the AGT in and out 3 times over the last few days and got the same thing happening several times in a row. I had a look at the soldered components (RF switch, caps), and they look well soldered (I even tested the connections with multimeter). The only think I cannot check is the solder points under the GPS module itself - so I am starting to wonder if there is a dry / bad solder point under the GPS chip, which means that there is no reliable connection between the GPS chip and the antenna.

I have not had the time to investigate in more details the other failing / problematic AGTs.

My question: is there a possibility for you to increase the level of quality control on the AGT production, in particular since this is a non modular, expensive board? ...

FYI @gauteh , this is maybe similar to problems you have had in the past.

@nseidle @PaulZC (since you were tagged on the previous relatively similar issue a few years back on sparkfunX/Artemis_Global_Tracker#29 ).

[gauteh]

Yes, we have had two recent bad Artemis nanos. tor. 15. feb. 2024, 12:38 skrev JR ***@***.***>:

…

Hi, I am building a large series of OpenMetBuoys (35) built using the AGT. It looks like we are facing some quality problems with the AGT unfortunately :( . This is a bit similar to what we had reported on sparkfunX/Artemis_Global_Tracker#29 <sparkfunX/Artemis_Global_Tracker#29> . More specifically: - one of the AGTs could not be programmed over USB when I took it out of its protective bag. Upon visual inspection of the AGT, I quickly found the issue: this is on the chip between the USB-C connector and the artemis module: as you can see, there was a blob of solder that was making a short (picture from my smartphone, so not super good, but this is well visible): IMG_20240207_144823.jpg (view on web) <https://github.com/sparkfun/SparkFun_Artemis_Global_Tracker/assets/8382834/f946683e-8ea2-4460-a308-ff99462cc01c> luckily I was able to work with the finest tip of my soldering iron, and to clean this blob of solder. After this, the AGT works just fine and can be programmed. This is fine in the end, but for an less experienced end user this may look like a "bricked" AGT... - I have several issues with 6 other AGTs. It seems that they are working only intermittently. I did not have as much time to debug these as I wanted, but I looked into 1 in particular. The issue is intermittent: if I take the AGT out of my lab (20 degrees C) and put it outside, it works normally for a while. Then after a few hours out in the cold (-5 degrees C), the GPS becomes unable to get a fix, so I suppose that the GPS signal does not reach the GPS chip or something like this (I can see the blue GPS LED turn on, and the GPS communication is working fine, but there is no GPS fix obtained even after 20 minutes). This seems to be quite "reproducible": I took the AGT in and out 3 times over the last few days and got the same thing happening several times in a row. I had a look at the soldered components (RF switch, caps), and they look well soldered (I even tested the connections with multimeter). The only think I cannot check is the solder points under the GPS module itself - so I am starting to wonder if there is a dry / bad solder point under the GPS chip, which means that there is no reliable connection between the GPS chip and the antenna. I have not had the time to investigate in more details the other failing / problematic AGTs. My question: is there a possibility for you to increase the level of quality control on the AGT production, in particular since this is a non modular, expensive board? ... FYI @gauteh <https://github.com/gauteh> , this is maybe similar to problems you have had in the past. @nseidle <https://github.com/nseidle> @PaulZC <https://github.com/PaulZC> (since you were tagged on the previous relatively similar issue a few years back on sparkfunX/Artemis_Global_Tracker#29 <sparkfunX/Artemis_Global_Tracker#29> ). — Reply to this email directly, view it on GitHub <#8>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAN36ZTWVGNME6THMNEP5LYTXXU7AVCNFSM6AAAAABDKC36IKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGEZTMMZRGEYTEMI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jerabaul29]

What kind of issues did you have @gauteh ?

I think that @tnn77 also had some issues (he had some on the last batch) - may take some time to get an answer, he is on an icebreaker now.

[brhoff720]

@jerabaul29 Thanks for the feedback and the image! In talking with our QC and Ops team, we have a rough idea on what likely happened with the solder blob and are taking steps to ensure that does not happen again. Have you reached out to our service team about a replacement?

[Funda]

@jerabaul29 we will get you taken care of. Please submit a return request on our website and we can go from there: https://www.sparkfun.com/returns

I look forward to assisting you soon!

[jerabaul29]

Sorry for the delay of answer, I have been busy looking into this in more details :) .

Great to hear that you are able to find a QC solution for the solder blob issue @brhoff720 so that this gets less likely to happen again :) . I have been able to fix it myself and after removing the solder blob it works just fine, so no need to replace it, I could fix it since it was easy to find :) .

Thanks for the offer to help @Funda :) . So to summarize: the temperatures are a bit higher now (for the first round of testing it was down to -15 degrees Celsius outside, now it is barely -1 degree C at night), and all except 1 AGT are working well again. It is a bit annoying with the issues in earlier tests, but since the AGTs are going to be used in the Arctic during the melt season, it should be ok as we should not see too drastically low temperatures.

Regarding the 1 AGT that still has issues even at higher temperatures: it has been outside now for a couple of day, and it is really "fun": it works when temperature is above around 4 degrees C, and below that, the GPS stops working, probably a cracked or bad solder that does not sustain thermal expansion and breaks the circuit between the antenna and the GPS. What we will do is that we will deploy it in the open ocean a the limit of the Arctic, where temperatures are higher, then this should be ok.

So no need to replace the AGTs with temperatures difficulties for this time, given that the deployment will be in melting ice, it should not be too cold and work ok given the last round of tests. However, this is just luck on this specific deployment, and there are other deployments when we really need to be working well at -20 or -30 degrees C...

Looking into more details / doing more tests, it is rather clear that the issues are with contacts that get "bad" when temperature is too low, and I inspected the contacts I can see and they all look good, so this must be one of the contacts that are hidden under the GPS / Artemis and that I cannot see. If you have any way to enhance QC on future production rounds to make sure the hidden soldering points are well wetted and soldered, this would be very helpful :) . I feel quite sure of my diagnostic as this is also similar to some issues reported e.g. in sparkfunX/Artemis_Global_Tracker#29 .

I was discussing with some colleagues building other instruments recently, and the "tips and trick" they were giving is to prefer chip packages with the "small legs outside" kind of packaging, as they have found it quite a bit more robust in practice (guess the size and length of the small legs help a bit), and also easier to diagnostize and fix on ships without access to a lab.

I.e., the advice I received was to prefer packages that look like:

rather than

whenever possible. Not sure if this is something that can be kept in the back of the head when designing future AGT-like things (@PaulZC you were one of the designers of the AGT? :) )? :)