Truffle and Ganache providers, Metamask Provider Engine no longer accepted as valid by web3

[davidmurdoch]

Expected behavior

ganache-core's provider object should continue to be treated as a valid web3 provider.

Support was removed by 1c5f5c5#diff-e77bf80bc71ad9d4c1b43fc43f9f8635 and 30f720b#diff-e77bf80bc71ad9d4c1b43fc43f9f8635L69

Additionally, truffle-hdwallet-provider should also still be treated as a valid web3 provider.

Actual behavior

The following code use to work:

const provider = ganache.provider();
const web3 = new Web3(provider);

but now throws an Error: Please provide an valid Web3 provider error. Related issue at ganache-core: trufflesuite/ganache#290

Steps to reproduce the behavior

const provider = ganache.provider();
const web3 = new Web3(provider);

Error Logs

Error: Please provide an valid Web3 provider
    at ProviderResolver.resolve (/Users/adrianli/dev/repro-subscribe/node_modules/web3-providers/dist/web3-providers.cjs.js:720:13)
    at Web3.AbstractWeb3Module (/Users/adrianli/dev/repro-subscribe/node_modules/web3-core/dist/web3-core.cjs.js:27:51)
    at new Web3 (/Users/adrianli/dev/repro-subscribe/node_modules/web3/dist/web3.cjs.js:30:68)
    at main (/Users/adrianli/dev/repro-subscribe/index.js:13:16)
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at Function.Module.runMain (internal/modules/cjs/loader.js:744:11)
    at startup (internal/bootstrap/node.js:285:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:739:3)
Error: Please provide an valid Web3 provider
    at ProviderResolver.resolve (/Users/adrianli/dev/repro-subscribe/node_modules/web3-providers/dist/web3-providers.cjs.js:720:13)
    at Web3.AbstractWeb3Module (/Users/adrianli/dev/repro-subscribe/node_modules/web3-core/dist/web3-core.cjs.js:27:51)
    at new Web3 (/Users/adrianli/dev/repro-subscribe/node_modules/web3/dist/web3.cjs.js:30:68)
    at main (/Users/adrianli/dev/repro-subscribe/index.js:13:16)
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at Function.Module.runMain (internal/modules/cjs/loader.js:744:11)
    at startup (internal/bootstrap/node.js:285:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:739:3)

/Users/adrianli/dev/repro-subscribe/node_modules/solc/soljson.js:22
(Module.asmGlobalArg,Module.asmLibraryArg,buffer);var __GLOBAL__I_000101=Module["__GLOBAL__I_000101"]=asm["__GLOBAL__I_000101"];var __GLOBAL__sub_I_ABIFunctions_cpp=Module["__GLOBAL__sub_I_ABIFunctions_cpp"]=asm["__GLOBAL__sub_I_ABIFunctions_cpp"];var __GLOBAL__sub_I_ABI_cpp=Module["__GLOBAL__sub_I_ABI_cpp"]=asm["__GLOBAL__sub_I_ABI_cpp"];var __GLOBAL__sub_I_ASTJsonConverter_cpp=Module["__GLOBAL__sub_I_ASTJsonConverter_cpp"]=asm["__GLOBAL__sub_I_ASTJsonConverter_cpp"];var __GLOBAL__sub_I_AST_cpp=Module["__GLOBAL__sub_I_AST_cpp"]=asm["__GLOBAL__sub_I_AST_cpp"];var __GLOBAL__sub_I_ArrayUtils_cpp=Module["__GLOBAL__sub_I_ArrayUtils_cpp"]=asm["__GLOBAL__sub_I_ArrayUtils_cpp"];var __GLOBAL__sub_I_AsmAnalysis_cpp=Module["__GLOBAL__sub_I_AsmAnalysis_cpp"]=asm["__GLOBAL__sub_I_AsmAnalysis_cpp"];var __GLOBAL__sub_I_CompilerContext_cpp=Module["__GLOBAL__sub_I_CompilerContext_cpp"]=asm["__GLOBAL__sub_I_CompilerContext_cpp"];var __GLOBAL__sub_I_Compiler
abort({}). Build with -s ASSERTIONS=1 for more info.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! repro-subscribe@1.0.0 start: `node index.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the repro-subscribe@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/adrianli/.npm/_logs/2019-01-29T15_45_27_098Z-debug.log

Versions

Tested against web3@1.0.0-beta.41

[sirpy]

This is result of version beta-38
It prevents using any custom provider since it checks that input providers are http/socket etc... (known providers)
Truffle HDWalletProvider also no longer works

[jacbop]

So is this a bug, or a decision to halt support for the truffle hdwalletprovider?

[alcuadrado]

So is this a bug, or a decision to halt support for the truffle hdwalletprovider?

@nivida Has expressed multiple times that web3 1.0.0 will support EIP-1193 (Standard Ethereum Provider Interface), so this must be unintentional. The idea of that EIP is to be able to use any injected provider with any ethereum library.

[nivida]

I've added this error message because we didn't officially provide the possibility to customize Web3.js over the providers. I could offer to out comment the strict provider check until the stable release. After the stable release, you have to inject your own EIP-1193 compatible provider. I would also have a closer look on the Web3 Module API maybe it's the cleaner way for doing what you're doing :-)

[benjamincburns]

For those who might not know me, I'm part of the Truffle team, and @davidmurdoch and I oversee the development of ganache-core and ganache-cli.

we didn't officially provide the possibility to customize Web3.js over the providers

Official or not, the entire Ethereum Dapp development community relies on web3's support for custom providers. The fact that you'd break that support without first implementing deprecation warnings (ideally ones which indicate a specific timeline for the changeover) is severely concerning for us, to the point where we're considering switching Truffle away from web3.js in the near future. You might have announced this change somewhere ahead of time, but if so, we didn't see it in time to provide an apporpriate workaround for our users.

What are the future plans around stable release and roll out of breaking changes? Can we expect a deprecation mechanism will be included in the process of making breaking changes in the future? Will that deprecation process be used before the -beta.X flag is removed from the current version?

FWIW, we're big fans of EIP-1193's goals, but we're deeply concerned with how its roll out was managed in this case.

[benjamincburns]

We're also concerned with back compatibility with old versions of Truffle, Ganache and other software which is already in the wild with custom providers which don't implement EIP-1193. Is backward compatibility being considered?

[iurimatias]

@nivida This change is also a breaking change for several applications including Embark, Remix, Murmur, Status-js-desktop, etc.. As @benjamincburns said, using custom providers this way has always been something the community has relied on. Transitioning to EIP-1193 is fine but we need a deprecation warning and time to move existing tools to it.

Can you please consider rolling back this particular change?

[nivida]

@benjamincburns Thank you for your response.

The fact that you'd break that support without first implementing deprecation warnings (ideally ones which indicate a specific timeline for the changeover) is severely concerning for us, to the point where we're considering switching Truffle away from web3.js in the near future.

Sorry, this will never happen again.

What are the future plans around stable release and roll out of breaking changes? Can we expect a deprecation mechanism will be included in the process of making breaking changes in the future? Will that deprecation process be used before the -beta.X flag is removed from the current version?

The plan was and is to inform the community about breaking changes and to add warnings. Yes, the deprecation mechanism will be included in the process. This deprecation process will definitely used before the stable release happens.

We're also concerned with back compatibility with old versions of Truffle, Ganache and other software which is already in the wild with custom providers which don't implement EIP-1193. Is backward compatibility being considered?

Backwards compatible can't be achieved that easy because of the new architecture. But I'll check this closer on Monday. I'm currently with fever in my bed.

@iurimatias Yes, I will remove the strict validation. But the internal provider interface did also change.

If someone would like to create a custom provider then please have a look at this abstract class:
https://github.com/ethereum/web3.js/blob/1.0/packages/web3-providers/lib/providers/AbstractSocketProvider.js

[nivida]

Maybe we should create a Discord server as a communication platform where I could reach out all of you (Truffle, Ganache, Embark etc.). @benjamincburns @davidmurdoch @iurimatias

PS: I‘ve tried to contact one of the truffle developers over gitter but didn‘t got any response.

[benjamincburns]

I love the idea of having a dedicated channel we're all responsible for monitoring closely. I'm not a Discord user today, but I'll happily change that if others mentioned are regular users. Alternatively, maybe web3 needs a weekly or biweekly call similar to the AllCoreDevs call?

Either way, I also reached out to you on Gitter via DM and tried tracking you down in two channels about this issue and didn't get a response, so I think Gitter would be a poor choice.

But I'll check this closer on Monday. I'm currently with fever in my bed.

As important as this is, it isn't an issue that can't wait for your health. My & David's availability will be limited next week, but we'll keep an eye out this issue for further comms from you. In the mean time if you're happy with removing the strict provider type checks we can commit to submitting a PR (likely tomorrow), and you can review and merge when you're feeling better.

Thanks for addressing this, and I hope you feel better soon!

[light edit as my original version had an unintentional argumentative tone]

[benjamincburns]

But the internal provider interface did also change.

I missed this in my initial read through of your last message, @nivida. Just responding to say that we'll definitely want to discuss this further next week when you're feeling better. I think @iurimatias and others would agree with me that goal right now should be to

1. Restore support for existing custom providers in web3, and
2. Work together on a plan to roll out EIP-1193 in a less intrusive way

If removing the checks I mentioned isn't enough to achieve the first goal now, then we won't really gain anything by making that change and doing nothing else.

[nivida]

I love the idea of having a dedicated channel we're all responsible for monitoring closely. I'm not a Discord user today, but I'll happily change that if others mentioned are regular users. Alternatively, maybe web3 needs a weekly or biweekly call similar to the AllCoreDevs call?

Either way, I also reached out to you on Gitter via DM and tried tracking you down in two channels about this issue and didn't get a response, so I think Gitter would be a poor choice.

Yes, as we see gitter is definitely not the right platform for us. I think broader discussions about the public API and internals of the Web3.js lib should be discussed on GitHub. But I think for short questions etc. would a Web3.js slack server be good. @gnidan texted me the truffle team is using slack.

As important as this is, it isn't an issue that can't wait for your health. My & David's availability will be limited next week, but we'll keep an eye out this issue for further comms from you. In the meantime, if you're happy with removing the strict provider type checks we can commit to submitting a PR (likely tomorrow), and you can review and merge when you're feeling better.

Thanks for your understanding!
It's okay for me. I will do this changes quickly by my self. Thanks for offering this help.

I missed this in my initial read-through of your last message, @nivida. Just responding to say that we'll definitely want to discuss this further next week when you're feeling better. I think @iurimatias and others would agree with me that goal right now should be to

1. Restore support for existing custom providers in web3, and
2. Work together on a plan to roll out EIP-1193 in a less intrusive way

If removing the checks I mentioned isn't enough to achieve the first goal now, then we won't really gain anything by making that change and doing nothing else.

For restoring the support of existing custom providers do I have to write for Truffle, Ganache etc. an adapter otherwise the providers have to get updated.

For the rollout plan of the EthereumProvider does already exists the "Provider Ring".
Discord.
Ethereum Magicians - working group

[jaikrish5]

i found a solution to this by using
npm install web3@1.0.0-beta.37
thank u

[filips123]

@jaikrish5 This is already used by latest Truffle versions, but it should be compatible even with newer Web3 versions.

[davidmurdoch]

We're scheduling a call with all interested JavaScript providers with the purpose of coming to a consensus on EIP 1193 as well as discuss a timetable for deprecation and roll out. If you are interested in being involved or just want to listen in on the call please respond to the thread here: https://ethereum-magicians.org/t/eip-1193-provider-ring-call-meeting/2647

[aks30498]

So..... I can't use truffle-hdwallet-provider withd web3? Because am just learning all of this stuff and in Stiphen Grider's tutorials he uses that and am getting one or other error with every version of web3... So am stuck.

[jaikrish5]

So..... I can't use truffle-hdwallet-provider withd web3? Because am just learning all of this stuff and in Stiphen Grider's tutorials he uses that and am getting one or other error with every version of web3... So am stuck.

just use web3 1.0.0-beta37

[aks30498]

No I can't use beta-37 I've tried beta 37 it shows nothing on web3.eth.getAccounts() rather no line after this is executed... I've asked it on stackoverflow and mentioned the problem with each version I tried
Can someone help me with the issue:
https://stackoverflow.com/questions/54885565/web3-eth-getaccounts-not-working-with-infura?noredirect=1#comment96550433_54885565

[molten-firescar96]

@jaikrish5 had an off-by-one error, web3 1.0.0-beta36 is the latest version working with other web3 providers. Try clearing your node_modules folder before installing new versions to make sure you are getting the right one.

[aks30498]

@jaikrish5 had an off-by-one error, web3 1.0.0-beta36 is the latest version working with other web3 providers. Try clearing your node_modules folder before installing new versions to make sure you are getting the right one.

beta-46: Cannot read property map() of undefined One answer on stackexchange says use beta-36 to fix this issue

beta-36 and beta-37: Nothing after the getAccounts() statement gets executed, blank screen.

beta-26: core.addProviders is not a function

I've checked package.json, the version gets changed there so it shouldn't require clearing node_modules and all i guess.
Please check my stackOverflow post whick i have linked in an eralier comment.
Btw can it be an issue with truffle version? I console logged the provider and confirmed that address array is there so i don't think that it is causing any issue

[EronWright]

Note that the typescript definitions don't yet support a custom provider. Here's a workaround in typescript:

import * as Ganache from 'ganache-core';
import Web3 from 'web3';

function test() {
    const ganache = Ganache.provider({});
    const web3 = new Web3(<any>ganache);
}