Increase discoverability of embedded crates on crates.io

[japaric]

Hey all,

Lately I have been thinking about how we can increase the discoverability of the
embedded ecosystem (tooling, crates, documentation, etc.). Part of that effort
is making it easier to find relevant crates on crates.io and that's why I have
opened this issue.

Background:

A categories feature landed recently (well, not that recently) on crates.io
(you can see list of current categories here) and I think we may be able to
leverage it to increase the visibility of embedded crates.

I have gone through all the crates collected in this survey and have
classified them according to whether they are using categories on crates.io or
not (see list at the bottom). Most crates are not using categories at all. The
crates that are using categories are using the following categories:

• command-line-utilities
• concurrency
• data-structures
• development-tools
• development-tools::debugging
• embedded
• hardware-support
• no-std

Now some questions:

Have categories helped you find a crate that you needed for embedded development
before? Or did you even know about them?

Do you think that categories on crates.io help with discoverability, or that
instead we should have a more curated list of crates for embedded development?

Do you think we should reach out to the authors of the embedded crates listed at
the bottom and encourage them to add relevant categories to their crates?

What categories do you think we should encourage people to use?

Are there categories that you think are missing?

Finally a request: If you are an author of embedded / no-std crates on crates.io
please add categories to your crates to increase their discoverability!

---

cc @posborne @jamesmunns @thejpster @vitiral

Has categories

bitflags
Categories: no-std

xargo
Categories: embedded development-tools command-line-utilities

log
Categories: development-tools::debugging

cortex-m
Categories: embedded no-std hardware-support

slog
Categories: development-tools::debugging

cortex-m-quickstart
Categories: embedded no-std

cortex-m-rtfm
Categories: embedded no-std concurrency

heapless
Categories: data-structures no-std

No categories

lazy_static

intrusive-collections

fixedvec

sysfs-gpio

i2cdev

spidev

log_buffer

teensy3

cpuio

alloc_buddy_simple

pic8259_simple

volatile_register

spin

linked_list_allocator

rlibc

x86

prusst

m

lm4f120

fel-cli

alloc-cortex-m

cortex-m-semihosting

byteorder

build-const

defrag

Doesn't apply

meta-rust
Not on crates.io

meta-rust-bin
Not on crates.io

compiler_builtins
Likely will never be on crates.io as it's part of the sysroot

bridge-rpc
Not on crates.io

[japaric]

My answers:

Have categories helped you find a crate that you needed for embedded development
before?

So far I have only used the search function. I don't know how much having
categories or not helps with the search ranking, but it would be nice to able to
refine the search using categories. For example: "search all crates in the
'no-std' category using the keyword 'foo'". It seems this hasn't been
implemented yet; see rust-lang/crates.io#491.

Do you think that categories on crates.io help with discoverability, or that
instead we should have a more curated list of crates for embedded development?

I think both would help. A curated list certainly would be better but it's going
to take time to create; categories can help in the meantime and require less
maintenance (assuming more people start using them).

Do you think we should reach out to the authors of the embedded crates listed
at the bottom and encourage them to add relevant categories to their crates?

Sure, let's annoy them. I have cc'ed some of them in this thread. I'll be adding
categories to my crates as well.

What categories do you think we should encourage people to use?

At least no-std, embedded and hardware-support.

Are there categories that you think are missing?

I think an ARM Cortex-M category would help. There are already a bunch of crates
in that space; mainly device crates generated using svd2rust. It would be great
to be able to list all of them on crates.io. That way people won't create a
device crate that already exists.

[vitiral]

What is the difference between categories and tags? I feel like there are now two separate things to search for (not great).

What would be best is something like "are we web yet?" for embedded. Even something like awesome rust would be helpful. But it can't just be a dumping ground, it has to be curated.

http://www.arewewebyet.org

https://github.com/rust-unofficial/awesome-rust

[vitiral]

What is the difference between categories and tags? I feel like there are now two separate things to search for (not great).

What would be best is something like "are we web yet?" for embedded. Even something like awesome rust would be helpful. But it can't just be a dumping ground, it has to be organized and curated.

Whatever the solution, there needs to be clear recommendations for different tasks (with discussion of pros and cons), and we need to have different categories. Some examples:

• tutorials
• heapless types
• communication
• memory management
• concurrency support
• chip api
• board api

http://www.arewewebyet.org

https://github.com/rust-unofficial/awesome-rust

[japaric]

What is the difference between categories and tags?

From the blog post linked above:

How Categories are Different from Keywords and Searching

Categories are meant to augment keywords rather than replace them. Categories facilitate browsing, while keywords assist more in searching.

The thing is that keywords are too free form so it's hard to use them as categories. For example: are no_std, no-std, no-stdlib, no_stdlib, no-standard-library, etc. supposed to be the same thing? OTOH there's a single canonical category for crates that work without the standard library: the no-std category. That facilitates searching.

What would be best is something like "are we web yet?" for embedded.

I'm working on that right now. I'll post my draft on issue #15 later.

[lnicola]

There was some concern in #39 about the discoverability of crates supporting compatible, or mostly-compatible sensors. My proposal would be to pick a well-known device in the family, and use the description and keywords manifest elements to include the others. These should be quite visible in the crates.io search results.

I do have one concern, though, about the five keywords limit.

cc @hannobraun, @wose

[dbrgn]

I think it would be better to have crates for the actual device families, even if they're very similar.

So in the case of the SHT2x compatible temperature/humidity sensors, I'd prefer to have separate crates called si70xx and sht2x. They could both be shims on top of a more generic crate though.

In any case, when I'm looking for a driver for an SHT21 sensor (released in 2009 if I'm not mistaken), it would feel strange to use a crate named after a derivative product released around 2014 that happens to use mostly the same protocol 🙂 I feel that separately named crates would be better in this case (as long as the associated product is well known, of course it wouldn't make sense to create separate drivers for every HD44780 derivative...)

cc @wose

[hannobraun]

For the record, here's what I wrote in the other thread:

Idea: Create one crate per supported sensor, make their names extremely specific. Have all those crates consist of a single statement that re-exports everything from a weirdly named master crate.

(Not sure myself how serious I am with this suggestion. Seems a bit spammy, but definitely helps with discoverability.)

@lnicola I think you're not wrong about the description and keywords, and that might be a good way to convey the information in many cases. However, even though the description is visible in search results, it's less emphasized than the crate name.

For someone carefully reading all the search results, the description is a good way to convey what sensors are supported. But that still means lots of people who are inattentive, or in a rush, or overwhelmed, will miss the crate.

But to be clear, I don't want to convince everyone of the one true way of doing things. I'm just pointing out another option of releasing your work. Every situation is different, and each driver author can make the decision themselves.

Question to the audience: Does it make sense to write a Driver Development Guide (maybe as part of the embedded-hal documentation) which, on top of explaining the relationships between embedded-hal/implementations/drivers, has a chapter about publishing, which mentions the various possibilities with their pros and cons?

[therealprof]

@hannobraun Yes, I think we should definitely have that. There're quite a few pitfalls you most likely will come across when developing a driver and it's very likely others have encountered it before and developed a workaround.

[dbrgn]

@hannobraun yes, such a guide would be fantastic.

[therealprof]

I just noticed that we only have 6 properly tagged drivers on crates.io. 🤔 Seems the situation is even worse than I thought.

[wose]

After thinking some more about this: the amateur embedded developer, the hacker who uses what he has at hand, probably won't know of similarities between his device and some other, which happens to have an embedded-hal-driver implementation. And this is the main target group, right? To make it easier for them it would indeed be best to have a driver per device family, which name does give a clear hint on the supported devices (si702x or htu2xd...).

@dbrgn Let's try making our drivers more or less /feature complete/ and see if it's worth it to extract some of it in a common crate which both will use.

@hannobraun Yes, this would be awesome.

@therealprof I think there are just this 6 drivers already released to crates.io. So there probably aren't any wrong or non-tagged drivers.

[tones111]

As a hobbyist that's trying to get back in the game I've found getting my bearings within the rust embedded space to be much more confusing then I'd hoped. japaric's blog renewed my interest, but the crates.io/github landscape is a mess.

I understand the role each of each level in the stack, but trying to find crates for a particular layer and comparing against the competition is difficult. Two primary issues are a lack in naming consistency and that once something is published it's there forever. This makes searching near impossible.

As a concreat example:
stm32.rs - a deprecated book
stm32 - points to a "stm32-unified-crate" repo
stm32-rs - Really a parent repo to many published crates
STM32F0, STM32F1, STM32F2, STM32F3, STM32F4, STM32F7, STM32L0, STM32L1, STM32L4, STM32H7
But, these could have all been provided under one crate. But I don't believe there's a way to retract the names.
stm32f042 - Not to be confused with the stm32-rs children.
stm32f0x0 - This is getting rediculous
f3 - STM32F3 Discovery BSP
and on, and on, and on...

While part of what renewed my interest in getting back into embedded programming is the motivation and interest rust and japaric have been driving I think more care needs to be taken as a community to try and provide a naming scheme that helps identify a crate's purpose and a well communicated stack hierarchy. Having a free for all, first to publish system that (I believe) is unable to retract and consolidate is damaging the brand.

[therealprof]

@tones111 I agree that just checking out crates.io doesn't yield a unified picture and while I'm also not exactly happy with that fact there's really nothing what we can change about that. Getting rid of the "first to publish" system as you seem to suggest would either mean gating the community or censorship.

Have you checked out https://github.com/rust-embedded/awesome-embedded-rust? That is the place where we try to consolidate and explain the available resources.

[andre-richter]

Im with @therealprof here, I think a list like awesome-embedded-rust is currently the only sane way to properly provide a maintained and user-updated list. It also has the benefit that we can display resources in an organized and hierarchical manner. There's more flexibility than crates.io can provide.

Also, Crates.io is bigger than the embedded-wg and it for sure also has benefits the way it is designed. Introducing content moderation here looks unfeasible as of now.

[tones111]

@therealprof and @andre-richter

I appoligize, my rant didn't convey my point very clearly. I think we're all in agreement. I don't want to propose any curation / sensorship model to crates.io (for the reasons you point out). Rather that the original intent of this thread proposes trying to use crates.io features to aid in the discovery of the embedded ecosystem and I feel that is not be a viable solution.

Awesome-embedded-rust looks like it could provide the additional moderation, quality control, and organization that embodies the intent of this thread. One exciting aspect of all this is the amount of experimentation going on and it will take time to see how the ecosystem settles down. Part of the challenge until then is discovering competitors to the things that are listed to make sure visibility and quality/consistency with the rest of the stack remain proportional.

I do want to thank everyone that has been participating in bringing up this ecosystem (especially @japaric for leading the way). It's an exciting time to be an embedded dev.