truffle not using latest web3 version

[SvenMeyer]

• I've asked for help in the Truffle Gitter before filing this issue.

---

Issue

truffle not using latest web3 version

I know there is a problem installing web3.js globally, only solution I have found was to install web3 locally for every project, but how do I resolve this for truffle?

Steps to Reproduce

$ truffle version

Expected Behavior

use web3.js 1.0.0-beta.55

Actual Results

Web3.js v1.0.0-beta.37

Environment

• Operating System: Arch Linux
  $ truffle version
  Warning: Both truffle-config.js and truffle.js were found. Using truffle-config.js.
  Truffle v5.0.25 (core: 5.0.25)
  Solidity v0.5.0 (solc-js)
  Node v10.16.0
  Web3.js v1.0.0-beta.37

[wbt]

Web3 abandoned semantic versioning a while back and has been releasing breaking changes on the sub-patch beta number. Some of those changes break Truffle, and the core devs don't want to go up even one beta-version and figure out how to fix the breaking changes in that (even when it causes a security audit failure!) until they can handle ALL the breaking changes through to the latest version. The rate at which new breaking changes are introduced has been somewhat exceeding the rate at which they can be caught and handled, and the lack of semantic versioning for web3 makes managing that process harder.

As noted in the latest post in another discussion on the topic here, @eggplantzzz wrote "we are still getting some of this web3 stuff straightened out and should have a plan for moving forward on this in the near future. Sorry for not being able to say more with certainty but hang tight!"

That was a couple weeks ago. @gnidan wrote a week before that "There's a new plan now. Stay tuned."

There have not been updates posted since. Words like "asap" and "soon" take on a different meaning in this project than in other software projects, so you have to keep that in mind when reading these threads.

It's also important to remember that Truffle is not a truly open project: many decisions about its development are made in closed-door in-person meetings where the discussion and rationale for decisions is not shared, hence why those individuals are "not able to say more" and why any actual timeline for updating is anybody's guess and rightly viewed with a healthy dose of skepticism.

I would move to close this issue as a duplicate of one or more of the related issues and/or pull requests.

[eggplantzzz]

So Web3 decided to release version 1.0.0-beta.37 as version 1 stable. Here is the PR for that. After version 1 is released then maintenance will proceed and a lot of the current issues with version 1.0.0-beta.37 can get resolved.

And sorry @wbt, Truffle doesn't intend on being secretive about its development process. With Web3, however, we've been trying to figure out a way to move forward and trying to work with them. It hasn't been so long since the decision was made to release beta.37 as stable and we were hesitant to share more info as we've been expecting an official announcement (I'm not sure when exactly). Anyways, sorry if you felt like we were keeping you in the dark. It was not intentional!

[wbt]

@eggplantzzz Thanks for posting that PR link.

The frustrating points around the closed-door discussions is not just in the links above, but other places in the project too. As a handy example, here is one where there's an issue in the code that is definitely confusing & frustrating for newcomers. I proposed multiple fixes and even offered a pull request that the frustrated newcomers report solves the problem as anticipated. The commit leading to this specific issue was an outcome of "we are going to be having a discussion tomorrow" and had to be added even though I disagreed for good reason, as reported by those frustrated newcomers. I'm still in the dark about why a commit introducing a bug that makes things harder for newcomers was considered to be the best path forward or what net positive it's supposed to have. Good reason from someone not in the room when those decisions are made does not seem to be enough when the discussion reached an opposite conclusion for unknown/inarticulated or inaccurate reasons. (That point was also demonstrated in the policy decision that "Adjustments to gas costs are not breaking changes," even when the changes are demonstrated to be breaking existing code and even after that was the source of delaying the Constantinople update.)

The response was:

We will review this and come up with next steps from our perspective and discuss them here.

Over two months and a week later, I asked:

How have those closed-door discussions gone? Can we implement one of the proposed solutions, as people are getting confused (probably including many who can't find this particular page and the fork with a solution)?

I called that out again a couple days later but since these are closed-door discussions, only insiders know how they went. When I first learned about how open-source works, I was taught that any discussions in a more rapid synchronous channel (like an in-person meeting or teleconference) should be summarized on the archival listserv (now, it would be GitHub Issues), including who was there, what decisions were made, the reasons for them, and reasons for an alternative path that were considered but overridden. I would like to see more of that happening in these projects.

I also learned about the general development pattern for open source in which a developer (a) forks a project, (b) makes changes to his/her local fork, (c) runs the code locally and iterates with (b) as needed until the code does something useful that it didn't do before or is otherwise a net improvement, then (d) commits changes to a local copy. The developer then (e) offers those changes back to the open source community, where it might be merged in with a "main" line that is the copy most frequently forked for step (a), or if whomever is responsible for maintaining that copy refuses and the change is too useful, becoming a copy available for use and forking by others in step (a). That model is not how things currently work in this ecosystem, even step (c) by non-insiders. Proposed forks and solutions of step (e) are rejected on inaccurate grounds about what will happen asap and perhaps a bit of NIH syndrome.

Too many of the decisions in this ecosystem appear to be made arbitrarily with costs imposed on a community as a primarily one-way process. Even the decision to just say "we're going to call beta37 the 1.0 stable release, despite all the good reasons why we've not considered it a stable release and despite all the important issues in it we've been finding and fixing over more than the past half year" seems like a somewhat arbitrary marketing-driven versioning decision and probably a bit of a lie on the "stable release" language which seems more likely meant to satisfy some VC or manager or PR purpose than to necessarily have the appropriate technical backing. (To me, changing to 1.0.0 could just be used to signify a breaking change; it doesn't have to be conflated with completion of a major milestone or even stability in the release, as seems to be conflated in Web3. Hyperledger Fabric put their first LTS release at whatever version number they were at when they thought they were ready, specifically 1.4).

[gnidan]

@wbt Firstly: thank you for calling a spade a spade.

If I may be candid, as you have, this whole business with Web3.js has been a significant challenge for all parties involved.

Let me provide some context here.

Truffle + Web3.js: a current history

Starting from v5.0.0 and until latest, Truffle has supported exactly one version of Web3.js: 1.0.0-beta.37. Web3.js is currently latest at 1.0.0-beta.55. Numerous attempts have been made to upgrade to later betas, but none of them have passed CI.

Since web3's beta.37, there have been huge refactoring efforts going on, in addition to bug fixes. The problem is, there have always been breaking changes. Some of these were breaking only to Truffle, some of them have also been breaking to Truffle users.

Comment notes / edits

• I write this comment to provide some perspective on this ongoing Truffle/Web3 dependency upgrade saga. I write from my perspective as an employee of Truffle Blockchain Group, as a stakeholder in the Ethereum/blockchain application development community, and from the perspective of Truffle Blockchain Group to the best of my ability to represent that.

• Disclaimer: my perspective is limited. The conversation around Web3.js's beta is happening in lots of places, and I have not kept up everywhere. I defer to others who may more aware of the technicals and current developments. I have mostly just served as a facilitator for this ongoing discussion.

• Edits include: section headers, copy improvements, interstitial corrections, clarifications.

Initial proposal: maintain separate Web3.js v1.0 / v2.0 development branches

The first notable breaking change was the introduction of EIP-1193, which took us here at Truffle by surprise, as well as other projects, I believe. As a result, someone started a Discord server to discuss this, posted about it on GH, and got a number of interested stakeholders to join. You can see this issue for where things kind of hit the fan.

Anyway, in that Discord, @iurimatias from Embark proposed the idea (paraphrasing) "release beta.37 as 1.0 stable, because that's what everyone is using". Basically, "stable" means "patchable", and the version of web3 that everyone uses needs to be patchable, for hotfixes and the like.

First approach: keep up with latest web3

It was not clear at the time that the proposal was the best option, since there was generally understood consensus among the maintainers of a number of projects (Web3, Truffle, Embark, Zeppelin, et al.) that we could work together to support EIP-1193 and other big changes. This was the plan for a number of months, as @nivida over on Web3 continued to work on the various architectural improvements with periodic communication about the 1193 spec and so on.

But with architectural improvements, came other necessary decisions, like changes to the return format, and others, which I won't get into here, since @wbt you participated in the conversation in this upgrade PR, which discussed some of the more recent problematic changes for Truffle.

In the course of all this, since writing software is hard and takes a long time to do, esp. when the software has an existing ecosystem, it became more and more urgent to form a plan. I lost a couple weeks where 60-80% of my attention went to these issues, fielding questions and the like. It got the point where @spalladino reached out to me to find out Truffle's plan, because Truffle sits between Zeppelin and Web3.js. And it became clear that getting Truffle to align to the latest beta, in a non-breaking release, was becoming increasingly untenable.

Finding success coordinating

A big part of the problem has been the lack of good communication signal related to this issue. Everyone's busy in this space. To get things done that you feel are important, effective communication is essential. It's a lot easier to have effective communication by reaching out to a project's maintainer, learning what they feel is important, seeing how much available attention they have at the moment, what they need to know about what you want to propose, etc. It's not ideal. It grosses me out, really. I'd rather be spending all my time in GH having conversations. But that's not always possible. Sometimes it's just so much more effective when you open a socket connection, so to speak.

And so that's what we did. Me? Well, I was a real nuisance. I basically pestered Sam about how important this problem is to Truffle and the community at large, and we eventually found common ground. Santiago helped me craft our arguments as Web3 community stakeholders. Eventually, we concluded the point about stable-meaning-patchable, and realized that it was win-win for everyone if we moved forward with a Web3.js 1.0, for bug fixes, starting at beta.37, and 2.0, for architectural changes, from what's on latest now. (Sorry for being a nuisance, Sam. I hope you know how much respect I have for the position you've been in through all this!)

What's the plan now?

Anyway, that was a month or so ago, and things have been moving, but it's taking time. There was a draft blog post that, last I heard, was awaiting EF review? I don't know. I'm spoiling all of this in this comment now. I was hoping to avoid that, and that's the reason for the prior "secrecy", because I felt that it was not Truffle's place to make these kinds of public statements. It's Web3's release plan, they should be the one to publicize it. Truffle is here only to support that.

As I understand it currently, the next step is that Sam's going to release 1.0 from beta.37 and then continue the beta series for 2.0. I imagine there's still going to be a coinciding formal announcement. From there, Truffle, Zeppelin, Nomic Labs, Embark, and others have volunteered to take up the bastion of 1.0 maintenance so that Sam may continue to work on architectural changes.

There's an active discussion channel on Truffle's paid Slack team, and we've invited a bunch of single channel guests to represent various projects. I don't like that this is private; but I do want this problem solved. I'm happy to invite anyone who wants to part of this transition-period discussion, in so far as we have single-channel guest accounts left to give (they're finite, alas).

Anyway, not sure what timeline we're looking at, and sorry to ruin the surprise, but I just recognize that the urgency around this issue has not gone away, and I think I can speak for everyone when I say that we're all extremely interested in getting this sorted out.

Notes in response

I hope this provides some context. I meant to get to your other points @wbt, but unfortunately I don't have time right now to edit this comment further. but maybe they are best moved to another issue. I'll just note here that we on Truffle's development team continue to work on process changes to support the community and contributions at large, and we're very interested in making headway improving this further. Thank you for your understanding and for your calling attention to specific areas of improvements.

[nivida]

Starting from v5.0.0 and until latest, Truffle has supported exactly one version of Web3.js: 1.0.0-beta.37.

Yes, Fabian pinged me about the coming 5.0 release of truffle and we made a release together especially for it.

But with architectural improvements, came other necessary decisions, like changes to the return format, and others, which I won't get into here, since @wbt you participated in the conversation in this upgrade PR, which discussed some of the more recent problematic changes for Truffle.

The drastic change we made was that contracts do return BigNumber objects instead of strings. The workaround would be to use the toString() method the BigNumber object ist providing for normal use cases but the truffle suite doesn't count as "normal use case" project.

@wbt @gnidan The release will be done tomorrow as described in the release PR of the web3.js repository and thanks also to all other involved contributors!

[gnidan]

Truffle is now using web3@latest! (released as part of Truffle v5.0.29)

Thanks everyone!

[cbruguera]

Slightly related question I have: Is it possible to specify an older version of web3 (e.g. 0.20.x) without having to downgrade truffle?

[wbt]

@cbruguera Others may add more detail, but the short answer is no. The Truffle code is pretty tightly tied in to the web3 api and neither one have been good about following semantic versioning.