Q: Wait...why are we building this again?
Check out the Why section
Q: Why building it with physical devices instead of just using emulators/simulators?
Let's look at the following points
- Customers: Do our customers use emulators or physical devices? We should test under real usage conditions which cannot always be provided by emulators and simulators.
- Limitations: Emulators are often inferior in performance because they need to mimic the hardware + software layer, making them slower to use than real devices. Simulators, while faster, as it only mimics the software layer, might behave different than what real devices do. Here's a complete showdown of such debate.
- Inventory: As a Telecom, we can easily access a lot of devices with data plans loaded, in fact we have 2 full drawers in TOR and VAN with testing devices, but they are not well known to team members, especially new team members. The tracking of those drawers is also poor, we often have devices missing as they are not properly logged. With Devicefarm, it(will) serves as a centralized inventory for teams to access the physical devices for testing purposes.
- Visual effects: Having the real devices easily visible in the office, flashing and showing products built (by running automated tests), has higher visual impact to promote mobile testing than emulators and simulators running on one's computer.
- It's just cooler Don't you think?
Q: Why hosting in-house instead of utilizing cloud services (Perfecto, Saucelabs testobject, Amazon device-farm, Google firebase)
- Resources: As mentioned in the previous question, we are a telecom, we have good access to physical devices. (We already have lots of devices in both Toronto and Vancouver Why not take advantage of such by building on top of what we have?
- Pricing, we cannot justify spending 300k per year on a hosted device lab just yet, without proving value and defining long-term scaling strategy.
Q: Is this only for Toronto?
It will be in both TOR and VAN.
We are currently piloting in TOR to promote the concept, gather feedback and make sure it provides value for teams. Once we have a good turnout, it will be much easier to obtain backing to establish devicefarm in Vancouver as well.
Follow up Q
If it's currently only in TOR, how can VAN use it?
A: Aside from the manual use cases such as sync browsing and physically testing with the devices, all other functionalities can be performed remotely (e.g: web or native app automation).
There's also a live stream broadcasting the wall mount 24/7 (for access, join #devicefarm on Slack)
Q: How much % of user devices does Devicefarm cover
Currently(08/17) with the 14 devices, ~ 70% based on Top 25 mobile devices card in DOMO
Q: How does it work from a tech perspective?
Long story short: Mac minis power devices as Appium nodes, connects to a Selenium grid/server, which handles requests and delegate to nodes based on desired capabilities.
Q: How can I start using devicefarm for running my automated tests?
For web automation, if you are using Nightwatch.js, or you are still on the Ruby/Cucumber automation stack that we had previously implemented(yes it's still supported), then you are in luck! You can point to the Selenium server dedicated for devicefarm in your Selenium config. Where is this Selenium server you ask? Checkout our starter-kit:e2e devicefarm config!
Q: Why Appium?
- Open source
- Supports both iOS and Android native or hybrid
- Most popular in market = higher chance support
- Good documentation
- Easily integrated with Selenium, the concepts(such as desired capabilities) are easily understood by folks with Selenium experience, which most testers have
Q: Why Mac minis?
- iOS automation needs macOS (the operating system that Macs runs)
- Mac minis are the cheapest among all Mac machines
Q: How many devices does one Mac mini support?
Reason being: Each mini has 4 USB slots, and we want to avoid using USB adapters to plug in more, which will introduce performance impact
FYI Perfecto (who specializes in physical device testing) recommends 2 devices per server for performance optimization. We will start with 4 and gauge the performance as we go along.
Q: How are the devices managed?
Software layer: For a short-term strategy, we currently just use selenium grid's console, as well as some shell scripts to manage the devices. For a long term strategy, a proper MDM (mobile device management) system needs to be established.
Hardware / infrastructure: Currently manually managed
Q: What security measures are there
- The devices are secured by leveraging Shopguard's locking unit which is also used in TELUS stores, these units are connected to a central alarm so that if any of the devices is detached, or any of the cables is cut, the alarm will go off.
- The Mac minis are secured inside a lockable area in the wall mount / fixture
- We have a Nestcam monitoring the wall fixture 24/7, with live streaming and video history capabilities.
- The Selenium server will be accessed via API tokens (WIP) much like how Saucelab's API endpoint functions.
Q: How many devices do you plan to have? How do you scale?
To start, we will have around 16-24 devices on the wall fixture (phase 1), once we prove out the value and identify the need, we can either
- Go to phase 2 to craft out the device cabinet, which will host around 50-60 devices.
- Seek out cloud services (Saucelabs's Testobject, Perfecto, Amazon devicefarm, Google firebase, etc) as a long term scaling strategy. As in-house hosting might not be maintainable or cost-effective.
Q: Why "Devicefarm:, not "Devicelab" or anything else?
The rationale is that a farm can be "grown" or scaled up, whereas a lab is perceived as a confined space and a static image, a farm can be nurtured by the inspiration of its farmers (us all) where as a lab is more of a ... ok I can't BS any longer, it really doesn't matter so call it whatever you like.
Q: I can't find the answer I want!
- Ask in #devicefarm on Slack
- Contact @Nintendot / Slack: @benexpress / Email: email@example.com
- Contact @@telus/digital-farmers