Thread for discussing docs and better introduction to Piston

[bvssvni]

See https://www.reddit.com/r/rust_gamedev/comments/5v5tvo/cant_find_any_up_to_date_piston_tutorials_help/

We need to update the docs so new people gets less confused. One source of confusion is the backend agnostic design and why some libraries use Gfx for 3D, so I'll start there:

• The Piston core is divided into input, window and event_loop. This is a basic abstraction for writing generic code. The "piston" library reexports these core libraries and is meant to be used in generic code for applications. The "piston_window" library is a convenience wrapper that uses a particular set of backends (Gfx + Glutin).
• Writing a backend agnostic 3D API is very hard, but luckily people started to working on this problem very early. This is maintained under the Gfx organization. It is still under heavy development.
• Some 3D libraries in the Piston ecosystem uses Gfx, partly because we wanted to get early feedback on design and needed a way to test it. Another reason is that if Gfx succeeds, then this code gets reusable. It is better to have an independent organization that focuses on 3D rendering, because it takes a lot of time to get these things right.
• Piston's core design does not depend on the window or graphics API. A lot of libraries developed under the PistonDevelopers organization can be used independently, e.g. the Image library or the Dyon scripting language.
• Piston's 2D API is independent on the window backend and designed for portability. It uses triangulation on the CPU by default, but this can be overridden by the backend. It uses generic code to allow intermediate layers, based on previous experience with 2D animation software. In the future we hope to get a software rasterizer backend and a graphics tree for pre-processing.

Long term goals for the Piston project

Piston has a goal to develop a fully fleshed game engine, but it will take several years to get there. In the meanwhile, we design libraries so people can use them as a framework. We are not trying to design everything ourselves, and use lots of libraries from the Rust ecosystem.

There are two kinds of activities performed under PistonDevelopers: Maintenance and research. The maintenance part is about designing, building and directing libraries and projects. The research part is about thinking outside the box, work on stuff that can have long term significance, and using mathematics to formalize ideas. Sometimes, these two kinds of activities overlap.

What makes the Piston project unique, is the special mix of people with varied interests that work on projects for their own reasons. We try to make this a stable environment for long term collaboration. Policies are designed using analysis when possible. There is a simple thumb rule "people who work on a project decide". This is working very well, because people have different opinions, but it is often hard to understand the full picture unless you are working on it. Everybody are given write access to everything, so you can easier switch to work on something else when you need to.

If you have an idea and want to make it into a community project, you are free to start it under PistonDevelopers. Sometimes people choose to move stuff to another organization, which is OK. You decide.

We have very high ambitions for this project, but also focuses on getting benefits along the way. This means that a lot of people working on Piston do it because they want to achieve a personal goal, or they are using libraries in other projects. There is no separation of groups, but people tend to focus their efforts on a few projects at a time.

So, what is the end goal? The Matrix, develop Safe Artificial Superintelligence, or become immortal and rule over the galaxy?

The answer is yes.

Writing "Idiomatic" code for Piston

The "idiomatic" way of writing Piston code is to use the model/view/controller pattern. This is a simple design that has been tested in the industry for many years. The good thing about this pattern is that it scales, so you can combine it with other patterns at lower levels.

• A model is simply some data structure, database etc.
• A view is how something is rendered.
• A controller stores the state, transforms input events into other events or actions, or deal with application logic in general.

Currently, there is no traits defined for this abstraction, it is simply a way of splitting up code into reusable parts.

Two simple examples of this pattern is timer_controller and button_controller. More libraries will come, but you can take a look here for an updated overview.

This code is reusable for any model or view, but it is also more work to set up. You might want to use a tailored API for your use cases, e.g. Conrod for UI. This is why Piston uses a modular design with a small core, so the code that is reused for particular projects, e.g. editors, can be worked on in parallel with other projects.

What we mean by "idiomatic" code is that this pattern does work and has a predictable maintenance cost for large projects. It is not meant to mean that you can't do any better or that everyone should program this way. The great thing about Rust is that it allows a wide style of designs, e.g. functional APIs, stream processing etc. You should not think that one particular flavor is superior to another, but learn different techniques and figure out the trade-offs. You know best what works for you! 😄

https://github.com/PistonDevelopers/piston/wiki/Piston-overview

[BedfordWest]

Thank you for this! I wish I had seen this when I first picked up Piston - it would have saved me a lot of headache trying to understand the backend, API, window, and event_loop organization. If I get the time, I may try to do some blogging about my experience and/or write up some documentation for this stuff.

[BedfordWest]

One problem I encountered quite a bit so far is referencing out-of-date posts, examples, or tutorials. When I first started looking at Piston, I went through the source code for some of the games referenced on the main Piston site, but many of them have older source code that is no longer being actively maintained. Additionally, several community posts and tutorials use fairly old versions of Piston - sometimes using even some of the deprecated libraries.

Given the relatively small amount of these resources and how commonly they show up in search engines or posts for Piston, it might be beneficial to do a FAQ-style reference against known resources and problems while the project is still in a state of relative flux.

[BedfordWest]

One point I only realized earlier today:
Glutin actually utilizes winit for the window creation and management. This was confusing to me for two reasons:

• window management was initially a built-in functionality of glutin
• glutin describes itself as a pure Rust alternative to GLFW but only references OpenGL in the README, whereas GLFW is working to support Vulkan

After discovering (rust-windowing/glutin#726) I realized winit was later branched off of glutin as a window creation and management library to support multiple backends.

I then discovered Vulkano, which utilizes vulkano-win for Vulkan window creation and management through winit. This was important for me to realize that utilizing piston_window as a wrapper really sets me up for using OpenGL as my back end with Gfx as the API. This was an important distinction, given my initial impression at a glance of Gfx + Glutin was that it could possibly support OpenGL or Vulkan as the backend without much custom configuration.

[NoraCodes]

I'm interested in maybe adapting https://silverwingedseraph.net/piston-a-game-library-in-rust and it's upcoming sequel to be includable in the library docs; is that something you'd be amenable to merging?

[bvssvni]

Perhaps put them here: https://github.com/pistondevelopers/piston-tutorials

[echochamber]

@bvssvni

I just read the discussion here (#1166) and it gave me a few ideas for docs/introduction improvements.

I think we should promote more visibility of "how to write idiomatic piston code" since I think it would give more confidence to new users when exploring piston, not just users new to piston but to also new to game/graphics programming altogether.

Some ideas/suggstions I've had for this.

(I'm happy to open some PRs for these updates if you are on board with them):

• I think the piston_examples crate is really useful for showing working examples, but the Readme doesn't explicitly state that these are working examples and not necessarily the "Idiomatic" way to write your own code since what is idiomatic depends on a few factors, such as if you're writing generic controller code or not (like you mention in the FAQ you linked). I'd update the Readme here to clarify this.
• Another potentially useful idea I had was to make the piston_examples crate have two "src" directories, one for actual working examples and another which is just "examples of the different ways to write idiomatic piston code, and when to use which idioms".
• Last idea: I think there should be an entire page in the Wiki for the various piston idioms and when to use what. It should be linked on the main piston crate's Readme since its one of those things that I think should be very visible. Or if not the main piston crate, definitely the piston_examples.

Let me know what you think of these

Second part, should we try making a piston book (similar to TRPL).

I think the Rust team does a phenomenal job of making learning the language as pleasant as possible. A huge part of this is how much time and effort goes into The Rust Programming Language (shout out to @steveklabnik). Do you think it would be good to start a long term project for collecting a bunch of this information into a book? The idea being, each chapter of the book would still be easily referable (just like the current wiki/docs design) but it would present newcomers with a single continuous reading path to follow to learn the the basic concepts of piston (remove the stress of "what should I read/learn next" which is a real).

If this is something you think would be a good long term project, I'd be willing to try starting it off. I'd still consider myself intermediate at best regarding Piston and Rust but if folks here would be willing to review my work I'd be happy to start the process (likely with a discussion on how to structure and order the book). One of my favorite part of writing docs is I get to learn by people much more knowledgeable than myself reviewing and correcting my mistakes in the docs!

[bvssvni]

@echochamber Sounds like good ideas. I can help with collecting information.

Here is something I would like to have in a book (in addition to the things mentioned in this thread):

• Get/set design guidelines - The Rust guidelines are good, but not complete enough for Piston API design
• About writing window wrappers (generic window backends, custom event loops etc.)
• Writing generic applications
• Maintaining a large open source project (Eco)

I am currently working on a window wrapper for using OpenVR with raw OpenGL, so I can write some things about this. I'll edit this comment first, then put it on the wiki later (or a book if we decide to do that).

About writing window wrappers (work in progress)

Piston uses a small core to make it easier to integrate generic libraries. This can sometimes lead to a lot of boilerplate when setting up a specific backend. To make it easy to start new projects and write examples and demos, it is common to write a "window wrapper" that uses the traits in the Piston core to extend a window backend with extra functionality.

For example, when programming for Virtual Reality (VR) you need to create extra buffers and render 3 times: Left eye, right eye, and a desktop window. OpenVR uses a special event loop that syncs tracked devices at certain intervals, and this logic must be mapped back into the event model that Piston libraries use. In addition there is information that all VR applications need, such as the matrix transform of the head mount display. Writing up this each time you need a new application takes a lot of time. This functionality can be put in a window wrapper to reduce boilerplate.

Example of a window wrapper: Piston-Window.

• A window wrapper should implement Window, AdvancedWindow and BuildFromWindowSettings.
• A method draw_2d if it does custom rendering for Piston-Graphics
• A method draw_3d if it does custom 3D rendering
• A method next if it contains a default event loop
• Implement EventLoop if the event loop allows customization

The philosophy is to allow great flexibility in how you put building blocks together for your own tailored game engine. For new people the many choices can be a bit overwhelming, but experienced users tend to gravitate toward a specific setup, or chooses a backend that has the desired characteristics for a particular application domain. There is no window or graphics backend that is the perfect choice for every platform target or user pattern, so Piston tries to make this flexible by design.

Writing generic applications (work in progress)

Notice! This pattern is under development, so expect this to change as we get more experience with maintaining projects and make modifications to the design of Piston.

A generic application should be written as a library and not depend on a particular window backend or wrapper. If possible, it should not depend on a particular graphics API. This makes the code portable and composable (Piston's design allows multiple applications to be glued together, call other applications, or sometimes event "host" one application inside another).

Making your applications generic is a good exercise in writing reusable code. In many cases you find that some part of the code can be extracted as a library. A typical project often starts out as non-generic and is turned into generic after the code is starting to stabilize.

Generic applications often have in common that they handle events, call controllers and do rendering in a specific order. Normally, application logic and rendering is separated concerns, which is something you can exploit to make your code more reusable. For example, to target different platforms with native interfaces for a better user experience.

The recommend pattern to use is model/view/controller. Instead of thinking of your application as single object, you can often split it into 3 parts: Model (the data, e.g. database), controller (handling events and logic) and view (rendering).

1. If possible, you should write your application as a controller with a method event<E: GenericEvent>(&mut self, e: &E, ...) and do separate rendering. The reason to separate rendering from handling events, is that rendering often happens with custom setup for a specific backend. This part of the code has more dependencies on the platform, while the application logic is often independent of hardware, framework or the operating system.
2. There are multiple ways you can support custom rendering. For example, you can return a list of high level elements to render using an enum and include a Cargo feature for rendering using a default backend. You can also render 2D using Piston-Graphics or 3D using Gfx to target multiple graphics APIs such as OpenGL, DirectX, Metal, etc.
3. When possible, separate the state needed for rendering into a View struct. This should be decoupled from the application controller, such that a different View can be written when needed.
4. You can use Input::Custom with a custom event trait to pass events from a custom interface to your application controller through the event method. By talking through this "channel", there is only one line of code necessary to connect to the controller.
5. If your application need to pass events back to the interface, you can push them on a Vec stored inside the controller and iterate through them after calling event. This list can be cleared at the beginning of event.

Alternatively, if you can not write your application logic as a controller, you can create a function run that does the whole event loop, but is generic over window and graphics backend. To make rendering work with different backends, you can use a closure like this:

// dyon_generactors library
pub fn run<W, F, G>(
    script: &str,
    mut settings: Settings,
    prelude: Option<Module>,
    window: &mut W,
    draw_2d: &mut F
)
    where W: AdvancedWindow,
          F: FnMut(Viewport, &mut FnMut(Context, &mut G)),
          G: Graphics
{
    ...
    while let Some(e) = events.next(&mut window) {
        ...
        if let Some(args) = e.render_args() {
            draw_2d(args.viewport(), &mut |c, g| {
                ...
            });
        }
    }
}

The partial example above is from an application which works like a game engine, controlled by a Dyon script file to generate content. This way you can make your application even more flexible. It is used like this:

fn main() {
    let opengl = OpenGL::V3_2;
    let settings = WindowSettings::new("generators", [512; 2])
        .exit_on_esc(true)
        .opengl(opengl)
        .samples(4);
    let mut window: Sdl2Window = settings
        .build()
        .unwrap();

    let mut gl = GlGraphics::new(opengl);
    let settings = dyon_generators::Settings::new();
    dyon_generators::run(
        "examples/test.dyon",
        settings,
        None,
        &mut window,
        &mut |viewport, f| gl.draw(viewport, f)
    );
}

Notice how flexible this is:

• The application works for any window/graphics backend
• The user can change how the application behaves through scripting, e.g. import data from the web or other files

Instead of having one application which you configure each time you start it, you can use your application as a library and set up a tailored program for each use case you have. Of course, you can also ship binaries to users if you need to.

Many people believe that writing generic applications is hard. This does not have to be true, when you learn 1) how to think of your code in terms of the model/view/controller pattern and 2) how to use closures at application level.

Maintaining a large open source project (Eco) (work in progress)

In Rust code bases, it is often common to split functionality into small crates. This leads to a lot of Cargo.toml files to keep track of. Crates.io has many nice tools that can be added as as Cargo sub-commands, but these are often design for small, medium or large projects. What about super large projects??

For super large open source projects, like Piston, small problems like updating version numbers becomes humanly impossible to keep track of manually. What you worry about when maintaining the code, is whether the ecosystem is "consistent". Is every library updated to the latest breaking changes? Which version should they be updated? What if I don't want to update a specific version? Without this level of reasoning, there is no guarantee that there even exists a combination of library versions that suit a specific need. Users might get a project compiling with library A and B, B and C, but not A and B and C.

Luckily, we have written a tool for that: Eco!

Eco lets you define a config file that points to the raw Cargo.toml files online, such that you always get the latest data from where the development happens. It pulls the data, builds a dependency graph, and analyzes it for breaking changes. After a few seconds, it output a todo-list in JSON format.

[echochamber]

I made a new issue thread specifically for discussing what a piston book would look like, since I though posting that her might result in derailing other discussions that belong here but are not about the book. Would love to hear other people's thoughts there.

[bvssvni]

Research and Exploration

Most scientific research can roughly to be categorized into these two main categories:

1. Goal oriented research (e.g. invent a better GPU architecture)
2. Basic research (e.g. describe mathematically how a type of cell interacts with each other)

Over time, the results of basic research allows a wider range of goal oriented research. The importance of basic research is often underestimated, but it is clear to many people that it is important to at least some degree. Almost every aspect of modern civilization is a result of basic research long way back in time. It is hard to wrap your head around it when you grow up with some technology and don't think about all the work required to, not only develop it, but figure out the basic principles that allow that technology to exist in the first place.

For software and programming, there is a more blurry boundary between goal oriented and basic research. I find this analogy easier to use:

1. The space port (e.g. the design of Conrod helps you build UIs)
2. The rabbit hole (e.g. the ongoing research on path semantics)

"The space port" is an analogy of choosing where to go, depending on what your goals are. Imagine you could go to other planets, but to get there, you first need to visit the space port. These projects are vehicles for achieving stuff. There is work to do here, but we are pretty sure of what the end results will be. Some things can take you far, and might allow for infinite combinations, hence the analogue of space traveling. A port is a public place of meeting other people, which fits the analogue because such projects allow people to cooperate even when they have different goals.

"The rabbit hole" is a reference to Alice in Wonderland, where she enters what seems like a small tunnel and then tumble down into a whole large world on its own, that only seems to go deeper and deeper. Some research topics in the Piston project are of this kind. Yet, the reason it looks like a rabbit hole is because the topic is not well understood yet. The mystery feeling is a state of mind, not a state of the world. The reason we do this kind of research, is we believe it is important to live with a bit uncertainty and mystery. Over time, rabbit holes tend to either slowly turn directly into space ports or give rise to new ones as a byproduct. A rabbit hole can also lead to the discovery of new rabbit holes, for example in path semantics where new rabbit holes seems to appear all the time.

Example of a "space port" research project: Conrod

Conrod started out as a project around an idea of making a graph navigation UI framework, but soon narrowed the scope to an immediate UI. This was a good decision, because the result is a more powerful framework than the original idea.

While the overall goal of Conrod has always been clear, it has gone through lot of experimentation. At some point it used duck typing traits, at another point it used an Elm-inspired graphics framework on top of Piston-Graphics. As the features got more demanding, the concept of the widget state graph evolved. Lots of people have contributed to the Conrod project because it is immediately useful and easy to see how it can be improved.

What makes Conrod a "space port" project is by its known goal and the way it is used as a tool to achieve other goals.

Example of a "rabbit hole" research project: Path Semantics

Path semantics started as ideas for mathematical reasoning of APIs. The basic idea was that you could use functions to describe dependencies, and from this derive decisions about better API design.

While evolving this notation, it was discovered a useful way of writing called "associated functions". This was a way of expressing ideas that are generally hard to express in normal programming languages. Because it desirable that everybody understand these ideas the same way, e.g. when discussing API design, it meant the semantics of this notation needed clarification. The word "path" comes from the idea of "leading to somewhere" expressed in different ways, but also inspired by Homotopy Type Theory. Hence, the name "path semantics".

It turns out, that the notation that give rise to path semantics is suitable for theorem proving (which lead to more research goals). Expressions written in path semantical notation can be translated into equations and vice versa, but not every equation has a path semantical analogue, and it happens that those who have are very "meaningful". This is very interesting for the common sense problem in artificial intelligence (even more research goals). The problem is that path semantics is much more expressible than normal programming languages, so a grammar that cover all usages is hard to define. To study the grammar, a meta parsing library was developed, which is now used in Dyon and Eco. After a while it was realized that this would become a long term project, and was moved outside PistonDevelopers to not overburden people's inboxes with emails. Still, new major results are reported so people can follow the progress toward a new way of thinking about theorem proving and perhaps new kinds of programming languages.

What makes path semantics a "rabbit hole" project is the collection of related topics that are seemingly intractable, but with a core goal of getting a solid understanding.