Moddable SDK 4.2 breaks all of the code examples!

[kallistisoft]

Note that this bug report involves multiple issues, I will list them starting at the surface...

Build environment: Linux

Target device: Moddable Two used a reference, but affects all platforms

---

Issue 1 - Broken Manifest Files
Description

The changes in the $(MODDABLE)/examples/manifest_xxx files introduced in v4.2 break all of the examples; the only examples that (I'm assuming) aren't broken are the ones who's manifest files were updated in commit eea713c

The problem stems from the recent refactoring that resulted in changes to the examples/mainifest files, so that including $(MODDABLE)/examples/manifest_mod.json no longer references the timer code.

Steps to Reproduce
Compiling any of the book examples using the v4.2 SDK will result in a compilation error

# cc xsHosts.c (strings in flash)
In file included from .../moddable-public/xs/platforms/mc/xsHosts.c:43:
.../moddable-public/xs/platforms/mc/xsHosts.h:29:18: fatal error: modTimer.h: No such file or directory
   29 |         #include "modTimer.h"
      |                  ^~~~~~~~~~~~
compilation terminated.

Resolution
Adding "$(MODDABLE)/examples/manifest_base.json" to the includes of example manifest files resolves this issue. I would do some bash-foo and grep/sed the example manifest.json files and send you a pull request, but I don't have a framework in place for doing automated testing of the repo...

---

Issue 2 - PIU Examples no longer match the API
Description

There are multiple issues with the PIU based examples, as the last update to ch10-piu folder was ~2 years ago... Here I'm using the ch10-piu/helloworld as an example...

const textStyle = new Style({
	font: "24px Open Sans"
});

const sampleLabel = new Label(null, { 
	style: textStyle,
	string: "Hello, World",
	top: 0, bottom: 0, left: 0, right: 0
});

application.add(sampleLabel);

Issue (1) example.js vs main.js

Adding manifest_base.json will get the example to compile, but the program won't load do to a lack of a "main" module.

xsbug -> # break: (host): module "main" not found!

Issue (2) More manifest issues
Changing the file name to main.js and updating the modules section in the manifest will get the program to load but result in an undefined variable error due to the PIU module not being included...

xsbug -> helloworld/main.js (15) # Break: main: get Style: undefined variable!

Issue (3) Fundamental changes to the PIU API
Adding "$(MODDABLE)/examples/manifest_piu.json" to the include section fixes the missing symbols issue but the fundamental API pattern has changed so that 'application' is no longer a global symbol; as an aside I think this is a good design decision ;)

xsbug -> helloworld/main.js (25) # Break: main: get application: undefined variable!

...

Refactoring to use/export an Application object instance fixes that issues, but then the font isn't found as it hasn't been included in the resource section, etc...

Resolution
All of the PIU examples need to be refactored to work with the current version of the SDK, but the more fundamental issue is that the book is out of date... also the documentation on the website for piu is also out of date; see the sections prior to "Piu Object Reference"

As the book is supposed to be fundamentals guide to augment the online documentation, and as updating the PIU examples for the book repository would put it out of sync with the content of the book -- it would be helpful if some of the simpler PIU examples from the book were added to the collection of SDK examples to help round them out as the SDK examples tend to show higher-level concepts.

Hopefully you are working on a second edition of the book, but in the meantime it would seem like there should be a disclaimer with a link to the version of the SDK used in the examples.

I plan on keeping my copy as the book as I still find it to be a useful reference for the general architectural design decisions, but I unfortunately can't recommend it as a learning tool :(

[phoddie]

Thank you for your report. I think there may be some misunderstandings here. We tested examples from the book with Moddable SDK 4.2. I just ran a half dozen Piu examples after receiving your report, and all worked correctly.

Let me take a little time to carefully review your report and get back to you. Meanwhile, please withhold judgement.

[kallistisoft]

As always thank you for your prompt reply!

I think there may be some misunderstandings here.

I'm sure there is and it's probably on my side...

As a follow-up test I created an Ubuntu 22.04 VM and did a clean install of esp-idf/moddable and got the same results; the sdk examples run without issue but the book examples still fail while compiling due to modTimer.h not being found.

Meanwhile, please withhold judgement.

My apologies if the tone of my report was perceived as judgmental; I can assure you I'm rarely judgmental, but I am often ignorantly opinionated :)

[phoddie]

Let's try to work through these step-by-step.

But first, there's on important detail that you may have overlooked. Almost all the examples in the book (including all in Chapter 10) are mods. This is done so that they can be built and installed very quickly (1 or 2 seconds). Mods install so quickly because they depend on a host to provide the core modules they depend on, such as timer and Piu. The host is built and installed once using mcconfig. Once that is complete, the examples (mods) are built and installed using mcrun.

Page 10 of the book, titled "Running Examples," explains that each chapter uses a host to install the common code for the examples. It walks through installing a host using mcconfig followed by an example using mcrun on pages 12 and 13.

Page 413 of the book is titled "Installing the Piu host". It explains that first you must use mcconfig to install the host to run the examples. This is not optional.

The problem stems from the recent refactoring that resulted in changes to the examples/mainifest files, so that including $(MODDABLE)/examples/manifest_mod.json no longer references the timer code.

manifest_mod.json never included the timer module. Mods contain only JavaScript and the timer module is a native module. If manifest_mod.json included timer, mcrun would report an error.

Adding "$(MODDABLE)/examples/manifest_base.json" to the includes of example manifest files resolves this issue

I think you are trying to build the Piu examples using mcconfig. They are designed to be run only with mcrun.
Adding manifest_base.json to the mod's manifest may allow it to build with mcconfig however, it won't run. That's because it depends on the environment initialized by the host.

Changing the file name to main.js and updating the modules section in the manifest will get the program to load but result in an undefined variable error due to the PIU module not being included...

Piu is included in the host. The examples depend on the host being installed first.

Adding "$(MODDABLE)/examples/manifest_piu.json" to the include section fixes the missing symbols issue but the fundamental API pattern has changed so that 'application' is no longer a global symbol

The application global that the examples rely on is set-up by the Piu host when it calls new Application. If you build the examples using mcconfig, the host is not included, and therefore the application global is not created.

...also the documentation on the website for piu is also out of date; see the sections prior to "Piu Object Reference"

These are not full application examples; they are code fragments to illustrate specific points. They remain accurate. The Piu APIs have not changed.

---

Idea

Perhaps it would be helpful to improve error reporting in mcconfig. Earlier this year we made a change so that when native code is detected by mcrun, it reports a detailed error (this was motivated by a report from @tve). Here's the output for $MODDABLE/examples/helloworld when using mcrun instead of the expected mcconfig:

> cd $MODDABLE/examples/helloworld
> mcrun -d -m -p esp
Native code detected:
  $MODDABLE/modules/files/resource/Resource.c
  $MODDABLE/modules/base/time/esp/modTime.c
  $MODDABLE/modules/base/timer/modTimer.c
  $MODDABLE/modules/base/timer/mc/timer.c
  $MODDABLE/modules/files/preference/esp/modPreference.c
  $MODDABLE/modules/base/instrumentation/modInstrumentation.c
  $MODDABLE/modules/network/socket/lwip/modLwipSafe.c
Mods cannot contain native code. Did you intend to build using mcconfig?

We could do something similar for when mcconfig is used to build a mod (e.g. a project that should be built using mcrun). That should help diagnose the problem sooner. What do you think?

[kallistisoft]

Thank you for your detailed response! Your passion for your work is both obvious and admirable! I'm very happy that the problem (once again) was my ignorance :)

But first, there's on important detail that you may have overlooked. Almost all the examples in the book (including all in Chapter 10) are mods.

I absolutely overlooked that important detail! Installing the piu host and using mcrun to deploy the examples works perfectly, except for the multiple-screens example...

/modules/piu/All/piuContent.c (205) # Break: _create: skin is no object!

---

We could do something similar for when mcconfig is used to build a mod (e.g. a project that should be built using mcrun). That should help diagnose the problem sooner. What do you think?

I think that is an excellent idea!

Thank you again!

[phoddie]

Glad the problem is solved! Thanks for the quick confirmation.

I made a small addition to mcconfig to report an error if it looks like it is being used to build a mod. That'll roll out in the coming days. I hope it will save some developers some time.

Regarding multiple-screens, I found it works in the simulator (I can't try on a device at the moment). Here's what I did (make sure not to quit the simulator after running mcconfig so that the host stays loaded for the example built by mcrun):

cd $IO-PRODUCT-DEV-BOOK/ch10-piu/
mcconfig -d -m ./host/manifest.json
mcrun -d -m ./multiple-screens/manifest.json

If you are still having problems, please open a separate report with relevant details (what device you are using, does it work in the simulator, etc).

[kallistisoft]

Tested the multiple-screens demo on a fresh Ubuntu VM and it worked properly on both the simulator and Moddable Two... Deleting and re-cloning the book repo fixed the issue on my main machine.

[phoddie]

Phew.